{"record":{"id":"1f15cfd883cc6099","repo":"sgl-project/sglang","slug":"failed-to-load-the-tokenizer-if-the-tokenizer-is","errorCode":null,"errorMessage":"Failed to load the tokenizer. If the tokenizer is a custom tokenizer not yet available in the HuggingFace transformers library, consider setting `trust_remote_code=True` in LLM or using the `--trust-remote-code` flag in the CLI.","messagePattern":"Failed to load the tokenizer\\. If the tokenizer is a custom tokenizer not yet available in the HuggingFace transformers library, consider setting `trust_remote_code=True` in LLM or using the `--trust-remote-code` flag in the CLI\\.","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/utils/hf_transformers/tokenizer.py","lineNumber":208,"sourceCode":"        # MistralCommon tokenizers reject standard HF kwargs like\n        # trust_remote_code, use_fast etc. Retry without them.\n        if \"are not supported by\" in str(e) and \"MistralCommon\" in str(e):\n            return retry_without_mistral_common_kwargs(\n                tokenizer_name, *args, **common_kwargs\n            )\n        # If the error pertains to the tokenizer class not existing or not\n        # currently being imported, suggest using the --trust-remote-code flag.\n        if not common_kwargs.get(\"trust_remote_code\") and (\n            \"does not exist or is not currently imported.\" in str(e)\n            or \"requires you to execute the tokenizer file\" in str(e)\n        ):\n            err_msg = (\n                \"Failed to load the tokenizer. If the tokenizer is a custom \"\n                \"tokenizer not yet available in the HuggingFace transformers \"\n                \"library, consider setting `trust_remote_code=True` in LLM \"\n                \"or using the `--trust-remote-code` flag in the CLI.\"\n            )\n            raise RuntimeError(err_msg) from e\n        raise\n\n\ndef _resolve_tokenizers_backend(tokenizer_name, *args, **common_kwargs):\n    \"\"\"Resolve generic ``TokenizersBackend`` to a proper tokenizer class.\n\n    In transformers v5, ``AutoTokenizer`` falls back to ``TokenizersBackend``\n    when the model_type has no tokenizer mapping.  This retries with\n    ``use_fast=False``, then attempts loading by the class declared in\n    ``tokenizer_config.json``.  May still return a ``TokenizersBackend``\n    if all retries fail (with a warning).\n    \"\"\"\n    logger.debug(\n        \"Tokenizer loaded as generic TokenizersBackend for %s, \"\n        \"retrying with use_fast=False\",\n        tokenizer_name,\n    )\n    common_kwargs = {**common_kwargs, \"use_fast\": False}","sourceCodeStart":190,"sourceCodeEnd":226,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/utils/hf_transformers/tokenizer.py#L190-L226","documentation":"AutoTokenizer.from_pretrained failed with a non-recoverable error (not the auto-retried MistralCommon kwarg case): typically the tokenizer class can't be found because the model uses custom remote code and trust_remote_code was not enabled.","triggerScenarios":"Loading a tokenizer whose class lives in the model repo's remote code without --trust-remote-code / trust_remote_code=True.","commonSituations":"Custom or newly published tokenizers not yet merged into transformers.","solutions":["Set trust_remote_code=True (LLM(...trust_remote_code=True)) or --trust-remote-code on the CLI","Upgrade transformers so the tokenizer is natively supported","Verify the model path/revision actually contains tokenizer files"],"exampleFix":"# before\nllm = LLM(model=..., trust_remote_code=False)\n# after\nllm = LLM(model=..., trust_remote_code=True)","handlingStrategy":"try-catch","validationCode":"cfg = json.load(open(Path(model)/'tokenizer_config.json'))\nneeds_trust = 'auto_map' in cfg or cfg.get('tokenizer_class') not in KNOWN_HF_TOKENIZERS","typeGuard":null,"tryCatchPattern":"try:\n    AutoTokenizer.from_pretrained(name)\nexcept RuntimeError as e:\n    if 'trust_remote_code' in str(e): retry with trust_remote_code=True (only for trusted sources)","preventionTips":["Pass trust_remote_code=True for custom tokenizers from trusted repos","Upgrade transformers to gain native support"],"tags":["tokenizer","trust-remote-code","huggingface"],"backgroundTag":"tokenizer-load-failed","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}