gh-143658: importlib.metadata: Use str.translate to improve performance of importlib.metadata.Prepared.normalized

[hugovk]

We can apply @henryiii's improvement to packaging in pypa/packaging#1030 (see also https://iscinumpy.dev/post/packaging-faster/) to improve the performance of canonicalize_name and make it ~3.7 times faster.

Benchmark

Run Prepared.normalize(n) on every name in PyPI:

# benchmark_names_stdlib.py
import sqlite3
import timeit
from importlib.metadata import Prepared

# Get data with:
# curl -L https://github.com/pypi-data/pypi-json-data/releases/download/latest/pypi-data.sqlite.gz | gzip -d > pypi-data.sqlite
# Or ues pre-cached files from:
# https://gist.github.com/hugovk/efdbee0620cc64df7b405b52cf0b6e42

CACHE_FILE = "/tmp/bench/names.txt"
DB_FILE = "/tmp/bench/pypi-data.sqlite"

try:
    with open(CACHE_FILE) as f:
        TEST_ALL_NAMES = [line.rstrip("\n") for line in f]
except FileNotFoundError:
    TEST_ALL_NAMES = []
    with sqlite3.connect(DB_FILE) as conn:
        with open(CACHE_FILE, "w") as cache:
            for (name,) in conn.execute("SELECT name FROM projects"):
                if name:
                    TEST_ALL_NAMES.append(name)
                    cache.write(name + "\n")


def bench():
    for n in TEST_ALL_NAMES:
        Prepared.normalize(n)


if __name__ == "__main__":
    print(f"Loaded {len(TEST_ALL_NAMES):,} names")
    t = timeit.timeit("bench()", globals=globals(), number=1)
    print(f"Time: {t:.4f} seconds")

Benchmark data can be found at https://gist.github.com/hugovk/efdbee0620cc64df7b405b52cf0b6e42

Before

With optimisations:

❯ ./python.exe benchmark_names_stdlib.py
Loaded 8,344,947 names
Time: 5.1483 seconds

After

❯ ./python.exe benchmark_names_stdlib.py
Loaded 8,344,947 names
Time: 1.3754 seconds

3.7 times faster.

• Issue: importlib.metadata: Use str.translate to improve performance of importlib.metadata.Prepared.normalized #143658

[picnixz]

Do we have tests actually? if not, maybe it'd be good to add some.

[johnslavik]

Small ideas

[hugovk]

See #144083 for a followup.

[jaraco]

I'd use fixtures.parameterize for the parameters.

[jaraco]

This change needs to be ported to importlib_metadata to avoid getting overridden when syncing changes from that repo. I'd have contributed this change there instead, because that project provides performance tests for measuring the effect of changes such as these.

[hugovk]

This change needs to be ported to importlib_metadata to avoid getting overridden when syncing changes from that repo.

Please see PR python/importlib_metadata#529, and let's fix the test parameters over there first.

I'd have contributed this change there instead, because that project provides performance tests for measuring the effect of changes such as these.

Good to know, will target there first in the future.