Skip to content

job_key drops Æ/Ø/Å, so Danish names collapse to unreadable and colliding keys #537

Open

Description

@ai-tvang

Problem

slugify() folds with NFKD and then encode("ascii", "ignore"). ø and æ have no NFKD decomposition, so they are deleted. å decomposes to a plus a ring, so it becomes a bare a. Danish company and title slugs lose letters and stop being readable, and different words can end up with the same slug: Ørsted becomes rsted, and different words collide: Ærø and Rø both become r.

The docstring at tools/job_key.py:84 and the test at tests/test_job_key.py:153-154 pin this behaviour on purpose ("keyed as rsted since the rule existed"). The concern there is re-keying existing state files. That is a real trade-off, but the folder name derived from the key (/apply, /outcome) is also what users see on disk, and the key is meant to be a pure function of the posting.

Reproduction

tools/job_key.py:66-72 (slugify), reached from make_key at tools/job_key.py:114:

$ python tools/job_key.py --company "Ørsted" --title "Direktør, Åben Æble"
rsted_direktr-aben-ble
$ python tools/job_key.py --company "Ærø" --title "x"
r_x
$ python tools/job_key.py --company "Rø" --title "x"
r_x

Both collision lines were run against the upstream/master version of tools/job_key.py: two different companies get the same key, so the second posting would be treated as already seen.

tests/test_job_key.py:153-154 asserts make_key("Mærsk", "Søfarende") == "mrsk_sfarende".

Suggestion

Transliterate the Danish letters before the ASCII fold, using the standard Danish romanisation: æ→ae, ø→oe, å→aa (and the uppercase forms). Apply it before NFKD, otherwise å is already split.

_TRANSLITERATE = str.maketrans({
    "æ": "ae", "ø": "oe", "å": "aa",
    "Æ": "Ae", "Ø": "Oe", "Å": "Aa",
})

def slugify(text: str) -> str:
    if not text:
        return ""
    text = unicodedata.normalize("NFC", str(text)).translate(_TRANSLITERATE)
    decomposed = unicodedata.normalize("NFKD", text)
    ascii_only = decomposed.encode("ascii", "ignore").decode("ascii")
    return _NON_SLUG.sub("-", ascii_only.lower()).strip("-")

Result: oersted_direktoer-aaben-aeble, maersk_soefarende.

Re-keying: existing seen_jobs.json entries with these letters would no longer match make_key. --audit already reports this as keys_not_matching_current_rule and never renames automatically, so the migration stays a user decision. The PR would update the two pinned test assertions and the docstring, and add cases for å and uppercase Æ/Ø/Å.

Happy to open the PR if this direction is acceptable.

Activity

  1. MadsLorentzen commented on Oct 15, 2026

    @MadsLorentzen
    Owner

    Agreed on the cause, with one change of scope: the rule that drops ø and æ drops every Latin letter that has no NFKD decomposition (ß, ł, đ, þ, ð, œ too), so the fix is a fold table for that class rather than a Danish transliteration, and å stays a because it already decomposes without losing a letter (taking å->aa would move keys that work today for no gain). The pin in the test and the docstring was about stability of live seen_jobs.json files, and that cost is real, so the PR pairs the fold with --audit --migrate: rewrite the drifted keys in place, deterministic, backup first, reported. CHANGELOG under Changed with the one-time command, and the test pin flips to the new expectation with a comment naming this issue. Send it in that shape and I will take it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions