{"record":{"id":"8c7dbeec321d8a6b","repo":"huggingface/transformers","slug":"unrecognized-tokenizer-name-should-be-one-of-lis","errorCode":null,"errorMessage":"Unrecognized tokenizer name, should be one of {list(TOKENIZER_CLASSES.keys())}.","messagePattern":"Unrecognized tokenizer name, should be one of (.+?)\\.","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/transformers/convert_slow_tokenizers_checkpoints_to_fast.py","lineNumber":50,"sourceCode":"for name in SLOW_TO_FAST_CONVERTERS:\n    # Special cases for tokenizers that don't have their own Fast tokenizer\n    if name == \"Phi3Tokenizer\":\n        tokenizer_class_name = \"LlamaTokenizerFast\"\n    elif name == \"ElectraTokenizer\":\n        tokenizer_class_name = \"BertTokenizerFast\"\n    else:\n        tokenizer_class_name = name + \"Fast\"\n\n    try:\n        TOKENIZER_CLASSES[name] = getattr(transformers, tokenizer_class_name)\n    except AttributeError:\n        # Skip tokenizers that don't have a Fast version\n        pass\n\n\ndef convert_slow_checkpoint_to_fast(tokenizer_name, checkpoint_name, dump_path, force_download):\n    if tokenizer_name is not None and tokenizer_name not in TOKENIZER_CLASSES:\n        raise ValueError(f\"Unrecognized tokenizer name, should be one of {list(TOKENIZER_CLASSES.keys())}.\")\n\n    if tokenizer_name is None:\n        tokenizer_names = TOKENIZER_CLASSES\n    else:\n        tokenizer_names = {tokenizer_name: getattr(transformers, tokenizer_name + \"Fast\")}\n\n    logger.info(f\"Loading tokenizer classes: {tokenizer_names}\")\n\n    for tokenizer_name in tokenizer_names:\n        tokenizer_class = TOKENIZER_CLASSES[tokenizer_name]\n\n        add_prefix = True\n        if checkpoint_name is None:\n            checkpoint_names = list(tokenizer_class.max_model_input_sizes.keys())\n        else:\n            checkpoint_names = [checkpoint_name]\n\n        logger.info(f\"For tokenizer {tokenizer_class.__class__.__name__} loading checkpoints: {checkpoint_names}\")","sourceCodeStart":32,"sourceCodeEnd":68,"githubUrl":"https://github.com/huggingface/transformers/blob/a597f974857b3d92939971296bc0deb93d33d780/src/transformers/convert_slow_tokenizers_checkpoints_to_fast.py#L32-L68","documentation":"ValueError from convert_slow_checkpoint_to_fast in the maintenance script convert_slow_tokenizers_checkpoints_to_fast.py: the --tokenizer_name argument is not a key in TOKENIZER_CLASSES (the script populates it from transformers' *Fast tokenizer class names). The tool only batch-converts checkpoints for known tokenizer families.","triggerScenarios":"Running the script with --tokenizer_name llama2 (lowercase, wrong) instead of LlamaTokenizer; or a name for a tokenizer that has no Fast class (the try/except silently skips those when building the dict).","commonSituations":"Running the repo's conversion tooling with informal names; contributing a new tokenizer family and forgetting the Fast class or the registration step.","solutions":["Pass the exact class base name, e.g. --tokenizer_name Llama (script appends 'Fast').","Print valid keys first: from transformers.convert_slow_tokenizers_checkpoints_to_fast import TOKENIZER_CLASSES.","If the tokenizer has no Fast class, it cannot be batch-converted by this script."],"exampleFix":"// before\npython -m transformers.convert_slow_tokenizers_checkpoints_to_fast --tokenizer_name bert  # ValueError\n\n// after\npython -m transformers.convert_slow_tokenizers_checkpoints_to_fast --tokenizer_name Bert","handlingStrategy":"validation","validationCode":"from transformers.convert_slow_tokenizers_checkpoints_to_fast import TOKENIZER_CLASSES\nif tokenizer_name is not None and tokenizer_name not in TOKENIZER_CLASSES:\n    raise SystemExit(f\"--tokenizer_name must be one of {sorted(TOKENIZER_CLASSES)}\")","typeGuard":"def is_known_tokenizer(name: str) -> bool:\n    from transformers.convert_slow_tokenizers_checkpoints_to_fast import TOKENIZER_CLASSES\n    return name in TOKENIZER_CLASSES","tryCatchPattern":null,"preventionTips":["Print the valid TOKENIZER_CLASSES keys in tooling usage text.","Use the class base name (e.g. 'Bert'), not lowercase aliases."],"tags":["tooling","tokenizer","conversion","cli"],"backgroundTag":null,"analyzedSha":"a597f974857b3d92939971296bc0deb93d33d780","analyzedAt":"2026-08-14T18:24:08.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}