huggingface/transformers · error · ValueError

Unrecognized tokenizer name, should be one of {list(TOKENIZE

Error message

Unrecognized tokenizer name, should be one of {list(TOKENIZER_CLASSES.keys())}.

What it means

ValueError from convert_slow_checkpoint_to_fast in the maintenance script convert_slow_tokenizers_checkpoints_to_fast.py: the --tokenizer_name argument is not a key in TOKENIZER_CLASSES (the script populates it from transformers' *Fast tokenizer class names). The tool only batch-converts checkpoints for known tokenizer families.

Source

Thrown at src/transformers/convert_slow_tokenizers_checkpoints_to_fast.py:50

for name in SLOW_TO_FAST_CONVERTERS:
    # Special cases for tokenizers that don't have their own Fast tokenizer
    if name == "Phi3Tokenizer":
        tokenizer_class_name = "LlamaTokenizerFast"
    elif name == "ElectraTokenizer":
        tokenizer_class_name = "BertTokenizerFast"
    else:
        tokenizer_class_name = name + "Fast"

    try:
        TOKENIZER_CLASSES[name] = getattr(transformers, tokenizer_class_name)
    except AttributeError:
        # Skip tokenizers that don't have a Fast version
        pass


def convert_slow_checkpoint_to_fast(tokenizer_name, checkpoint_name, dump_path, force_download):
    if tokenizer_name is not None and tokenizer_name not in TOKENIZER_CLASSES:
        raise ValueError(f"Unrecognized tokenizer name, should be one of {list(TOKENIZER_CLASSES.keys())}.")

    if tokenizer_name is None:
        tokenizer_names = TOKENIZER_CLASSES
    else:
        tokenizer_names = {tokenizer_name: getattr(transformers, tokenizer_name + "Fast")}

    logger.info(f"Loading tokenizer classes: {tokenizer_names}")

    for tokenizer_name in tokenizer_names:
        tokenizer_class = TOKENIZER_CLASSES[tokenizer_name]

        add_prefix = True
        if checkpoint_name is None:
            checkpoint_names = list(tokenizer_class.max_model_input_sizes.keys())
        else:
            checkpoint_names = [checkpoint_name]

        logger.info(f"For tokenizer {tokenizer_class.__class__.__name__} loading checkpoints: {checkpoint_names}")

View on GitHub (pinned to a597f97485)

Solutions

1. Pass the exact class base name, e.g. --tokenizer_name Llama (script appends 'Fast').
2. Print valid keys first: from transformers.convert_slow_tokenizers_checkpoints_to_fast import TOKENIZER_CLASSES.
3. If the tokenizer has no Fast class, it cannot be batch-converted by this script.

Example fix

// before
python -m transformers.convert_slow_tokenizers_checkpoints_to_fast --tokenizer_name bert  # ValueError

// after
python -m transformers.convert_slow_tokenizers_checkpoints_to_fast --tokenizer_name Bert

from transformers.convert_slow_tokenizers_checkpoints_to_fast import TOKENIZER_CLASSES
if tokenizer_name is not None and tokenizer_name not in TOKENIZER_CLASSES:
    raise SystemExit(f"--tokenizer_name must be one of {sorted(TOKENIZER_CLASSES)}")

def is_known_tokenizer(name: str) -> bool:
    from transformers.convert_slow_tokenizers_checkpoints_to_fast import TOKENIZER_CLASSES
    return name in TOKENIZER_CLASSES

When it happens

Trigger: Running the script with --tokenizer_name llama2 (lowercase, wrong) instead of LlamaTokenizer; or a name for a tokenizer that has no Fast class (the try/except silently skips those when building the dict).

Common situations: Running the repo's conversion tooling with informal names; contributing a new tokenizer family and forgetting the Fast class or the registration step.

Related errors

• {error_message} requires the protobuf library but it was not
• You're trying to run a `Unigram` model but you're file was t
• `tiktoken` is required to read a `tiktoken` file. Install it
• Converting from SentencePiece and Tiktoken failed, if a conv
• Invalid checkpoint path: '{checkpoint}' attempts to escape `

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/8c7dbeec321d8a6b. Report an issue: GitHub.