{"record":{"id":"f0cb64deb0beed7b","repo":"huggingface/tokenizers","slug":"is-not-a-known-unicode-normalizer-available-ar","errorCode":null,"errorMessage":"{} is not a known unicode normalizer. Available are {}","messagePattern":"(.+?) is not a known unicode normalizer\\. Available are (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"bindings/python/py_src/tokenizers/normalizers/__init__.py","lineNumber":25,"sourceCode":"NFKD = normalizers.NFKD\nNFC = normalizers.NFC\nNFKC = normalizers.NFKC\nSequence = normalizers.Sequence\nLowercase = normalizers.Lowercase\nPrepend = normalizers.Prepend\nStrip = normalizers.Strip\nStripAccents = normalizers.StripAccents\nNmt = normalizers.Nmt\nPrecompiled = normalizers.Precompiled\nReplace = normalizers.Replace\nByteLevel = normalizers.ByteLevel\n\nNORMALIZERS = {\"nfc\": NFC, \"nfd\": NFD, \"nfkc\": NFKC, \"nfkd\": NFKD}\n\n\ndef unicode_normalizer_from_str(normalizer: str) -> Normalizer:\n    if normalizer not in NORMALIZERS:\n        raise ValueError(\n            \"{} is not a known unicode normalizer. Available are {}\".format(normalizer, NORMALIZERS.keys())\n        )\n\n    return NORMALIZERS[normalizer]()\n","sourceCodeStart":7,"sourceCodeEnd":30,"githubUrl":"https://github.com/huggingface/tokenizers/blob/6cfd9d385ca0ed91c10b49f0ce97d02cfde1b607/bindings/python/py_src/tokenizers/normalizers/__init__.py#L7-L30","documentation":"tokenizers' Python bindings only support four unicode normalizers by name: nfc, nfd, nfkc, nfkd. `unicode_normalizer_from_str` looks the requested name up in the NORMALIZERS dict and raises ValueError when it is absent, listing the valid keys. Any other string (typo, wrong case, or a normalizer that exists in Rust but has no named mapping here) is rejected.","triggerScenarios":"Calling `normalizers.Unicode(normalizer=\"...\")` (which routes through unicode_normalizer_from_str) with any string not exactly one of 'nfc', 'nfd', 'nfkc', 'nfkd' — e.g. 'NFC' (uppercase), 'nfkcf', 'strip', or an empty string.","commonSituations":"Typing the normalizer name with wrong capitalization; assuming other UnicodeNormalizer variants (Nmt, Strip, StripAccents) are selectable by string name; copying a name from another NLP library with different naming conventions.","solutions":["Use one of the exact lowercase names: 'nfc', 'nfd', 'nfkc', or 'nfkd'.","For other normalizers, use their dedicated classes directly, e.g. normalizers.NFKC(), normalizers.StripAccents(), normalizers.Sequence([...]) instead of the string-based Unicode() constructor.","Check the error message's 'Available are' list to see the accepted keys and compare for typos/case."],"exampleFix":"// before\nnorm = tokenizers.normalizers.Unicode(\"NFC\")  # ValueError\n// after\nnorm = tokenizers.normalizers.NFKC()\n# or\nnorm = tokenizers.normalizers.Unicode(\"nfkc\")","handlingStrategy":"validation","validationCode":"ALLOWED = {\"nfc\", \"nfd\", \"nfkc\", \"nfkd\"}\nassert normalizer_name in ALLOWED, f\"{normalizer_name!r} not in {sorted(ALLOWED)}\"\nnorm = tokenizers.normalizers.Unicode(normalizer_name)","typeGuard":"def is_valid_normalizer(name) -> bool:\n    return isinstance(name, str) and name in {\"nfc\", \"nfd\", \"nfkc\", \"nfkd\"}","tryCatchPattern":"try:\n    norm = tokenizers.normalizers.Unicode(name)\nexcept ValueError as e:\n    logging.warning(\"Unknown normalizer %r, defaulting to NFKC\", name)\n    norm = tokenizers.normalizers.NFKC()","preventionTips":["Keep normalizer names lowercase and use the exact keys nfc/nfd/nfkc/nfkd.","Prefer explicit classes (normalizers.NFKC(), normalizers.StripAccents()) over string-based Unicode().","Validate user-supplied normalizer names against the allowed set before constructing."],"tags":["python","normalizer","invalid-enum-value","configuration"],"backgroundTag":"invalid-enum-value","analyzedSha":"6cfd9d385ca0ed91c10b49f0ce97d02cfde1b607","analyzedAt":"2026-09-09T11:43:25.027Z","contentChangedAt":"2026-09-09T11:43:25.027Z","schemaVersion":2},"datasetVersion":"2026-09-16T04:17:20.429Z"}