huggingface/tokenizers · error · ValueError

is not a known unicode normalizer. Available are

Error message

{} is not a known unicode normalizer. Available are {}

What it means

tokenizers' Python bindings only support four unicode normalizers by name: nfc, nfd, nfkc, nfkd. `unicode_normalizer_from_str` looks the requested name up in the NORMALIZERS dict and raises ValueError when it is absent, listing the valid keys. Any other string (typo, wrong case, or a normalizer that exists in Rust but has no named mapping here) is rejected.

Solutions

1. Use one of the exact lowercase names: 'nfc', 'nfd', 'nfkc', or 'nfkd'.
2. For other normalizers, use their dedicated classes directly, e.g. normalizers.NFKC(), normalizers.StripAccents(), normalizers.Sequence([...]) instead of the string-based Unicode() constructor.
3. Check the error message's 'Available are' list to see the accepted keys and compare for typos/case.

Example fix

// before
norm = tokenizers.normalizers.Unicode("NFC")  # ValueError
// after
norm = tokenizers.normalizers.NFKC()
# or
norm = tokenizers.normalizers.Unicode("nfkc")

ALLOWED = {"nfc", "nfd", "nfkc", "nfkd"}
assert normalizer_name in ALLOWED, f"{normalizer_name!r} not in {sorted(ALLOWED)}"
norm = tokenizers.normalizers.Unicode(normalizer_name)

def is_valid_normalizer(name) -> bool:
    return isinstance(name, str) and name in {"nfc", "nfd", "nfkc", "nfkd"}

try:
    norm = tokenizers.normalizers.Unicode(name)
except ValueError as e:
    logging.warning("Unknown normalizer %r, defaulting to NFKC", name)
    norm = tokenizers.normalizers.NFKC()

When it happens

Trigger: Calling `normalizers.Unicode(normalizer="...")` (which routes through unicode_normalizer_from_str) with any string not exactly one of 'nfc', 'nfd', 'nfkc', 'nfkd' — e.g. 'NFC' (uppercase), 'nfkcf', 'strip', or an empty string.

Common situations: Typing the normalizer name with wrong capitalization; assuming other UnicodeNormalizer variants (Nmt, Strip, StripAccents) are selectable by string name; copying a name from another NLP library with different naming conventions.

Understand the failure class

Background: Invalid enum value errors: "Unknown type", "Invalid scope", "must be one of" — when a string is not on the library's allowed list — this error's family across 23 libraries.

Related errors

• encode: `sequence` can't be `None`
• encode_batch: `inputs` can't be `None`
• async_encode_batch: `inputs` can't be `None`
• async_encode_batch_fast: `inputs` can't be `None`
• None input is not valid. Should be a list of integers.

AI-assisted analysis of huggingface/tokenizers@6cfd9d385c (2026-09-09). Data as JSON: /api/errors/f0cb64deb0beed7b. Report an issue: GitHub.

Appendix: source

Thrown at bindings/python/py_src/tokenizers/normalizers/__init__.py:25

NFKD = normalizers.NFKD
NFC = normalizers.NFC
NFKC = normalizers.NFKC
Sequence = normalizers.Sequence
Lowercase = normalizers.Lowercase
Prepend = normalizers.Prepend
Strip = normalizers.Strip
StripAccents = normalizers.StripAccents
Nmt = normalizers.Nmt
Precompiled = normalizers.Precompiled
Replace = normalizers.Replace
ByteLevel = normalizers.ByteLevel

NORMALIZERS = {"nfc": NFC, "nfd": NFD, "nfkc": NFKC, "nfkd": NFKD}


def unicode_normalizer_from_str(normalizer: str) -> Normalizer:
    if normalizer not in NORMALIZERS:
        raise ValueError(
            "{} is not a known unicode normalizer. Available are {}".format(normalizer, NORMALIZERS.keys())
        )

    return NORMALIZERS[normalizer]()

View on GitHub (pinned to 6cfd9d385c)