{"record":{"id":"7ebf97e5079fea65","repo":"sgl-project/sglang","slug":"failed-to-load-the-tokenizer-if-you-are-using-a-l","errorCode":null,"errorMessage":"Failed to load the tokenizer. If you are using a LLaMA V1 model consider using '{_FAST_LLAMA_TOKENIZER}' instead of the original tokenizer.","messagePattern":"Failed to load the tokenizer\\. If you are using a LLaMA V1 model consider using '(.+?)' instead of the original tokenizer\\.","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/utils/hf_transformers/tokenizer.py","lineNumber":188,"sourceCode":"        tokenizer_name = client.get_local_dir()\n\n    return tokenizer_name\n\n\ndef _auto_tokenizer_from_pretrained(tokenizer_name, *args, **common_kwargs):\n    \"\"\"Call ``AutoTokenizer.from_pretrained`` with error handling.\"\"\"\n    try:\n        tokenizer = AutoTokenizer.from_pretrained(\n            tokenizer_name, *args, **common_kwargs\n        )\n        return tokenizer\n    except TypeError as e:\n        err_msg = (\n            \"Failed to load the tokenizer. If you are using a LLaMA V1 model \"\n            f\"consider using '{_FAST_LLAMA_TOKENIZER}' instead of the \"\n            \"original tokenizer.\"\n        )\n        raise RuntimeError(err_msg) from e\n    except ValueError as e:\n        # MistralCommon tokenizers reject standard HF kwargs like\n        # trust_remote_code, use_fast etc. Retry without them.\n        if \"are not supported by\" in str(e) and \"MistralCommon\" in str(e):\n            return retry_without_mistral_common_kwargs(\n                tokenizer_name, *args, **common_kwargs\n            )\n        # If the error pertains to the tokenizer class not existing or not\n        # currently being imported, suggest using the --trust-remote-code flag.\n        if not common_kwargs.get(\"trust_remote_code\") and (\n            \"does not exist or is not currently imported.\" in str(e)\n            or \"requires you to execute the tokenizer file\" in str(e)\n        ):\n            err_msg = (\n                \"Failed to load the tokenizer. If the tokenizer is a custom \"\n                \"tokenizer not yet available in the HuggingFace transformers \"\n                \"library, consider setting `trust_remote_code=True` in LLM \"\n                \"or using the `--trust-remote-code` flag in the CLI.\"","sourceCodeStart":170,"sourceCodeEnd":206,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/utils/hf_transformers/tokenizer.py#L170-L206","documentation":"AutoTokenizer.from_pretrained raised TypeError while loading the tokenizer; the classic case is LLaMA V1 tokenizers whose slow loader has an incompatible signature. The hint suggests the fast (tokenizers-based) alternative.","triggerScenarios":"Loading an original LLaMA V1 HF repo whose tokenizer implementation raises TypeError with the current transformers version.","commonSituations":"Old LLaMA V1 models/met repos, or transformers API changes breaking old tokenizer classes.","solutions":["Use the fast tokenizer variant indicated by _FAST_LLAMA_TOKENIZER (e.g. --tokenizer-mode mistramask-free fast variant / hf-internal-testing/llama-fast-tokenizer style name)","Upgrade transformers so the tokenizer class signature matches","Re-point tokenizer to a compatible tokenizer.json-based repo"],"exampleFix":null,"handlingStrategy":"fallback","validationCode":"# pre-check tokenizer type from tokenizer_config.json before loading","typeGuard":null,"tryCatchPattern":"try:\n    AutoTokenizer.from_pretrained(name)\nexcept RuntimeError as e:\n    if 'FAST_LLAMA' in str(e) or 'LLaMA V1' in str(e): load the fast LLaMA tokenizer variant instead","preventionTips":["Prefer tokenizer.json (fast) repos for LLaMA V1","Keep transformers updated"],"tags":["tokenizer","huggingface","typeerror"],"backgroundTag":"tokenizer-load-failed","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}