{"record":{"id":"5c0d1a748a9260b8","repo":"sgl-project/sglang","slug":"fastokens-failed-to-load-tokenizer-for-tokenizer","errorCode":null,"errorMessage":"fastokens failed to load tokenizer for {tokenizer_name!r}. This model's tokenizer may not be supported by fastokens — see https://github.com/crusoecloud/fastokens. Re-run without --tokenizer-backend=fastokens to use the default backend.","messagePattern":"fastokens failed to load tokenizer for (.+?)\\. This model's tokenizer may not be supported by fastokens — see https://github\\.com/crusoecloud/fastokens\\. Re-run without --tokenizer-backend=fastokens to use the default backend\\.","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/utils/hf_transformers/tokenizer.py","lineNumber":556,"sourceCode":"            tokenizer = _auto_tokenizer_from_pretrained(\n                tokenizer_name, *args, **common_kwargs\n            )\n\n            # With fastokens, the patched TokenizersBackend.from_pretrained already\n            # returned a tokenizer whose backend is a fastokens shim. Re-resolving via\n            # the declared class (e.g. Qwen2Tokenizer) would discard that work.\n            if (\n                type(tokenizer).__name__ == _TOKENIZERS_BACKEND\n                and tokenizer_backend != \"fastokens\"\n            ):\n                tokenizer = _resolve_tokenizers_backend(\n                    tokenizer_name, *args, **common_kwargs\n                )\n\n        return _apply_post_load_fixes(tokenizer, tokenizer_name, tokenizer_revision)\n    except Exception as e:\n        if tokenizer_backend == \"fastokens\":\n            raise RuntimeError(\n                f\"fastokens failed to load tokenizer for {tokenizer_name!r}. \"\n                f\"This model's tokenizer may not be supported by fastokens — \"\n                f\"see https://github.com/crusoecloud/fastokens. \"\n                f\"Re-run without --tokenizer-backend=fastokens to use the default backend.\"\n            ) from e\n        raise\n\n\n# ---------------------------------------------------------------------------\n# Exported helpers (used by processor.py, etc.)\n# ---------------------------------------------------------------------------\n\n\ndef _fix_added_tokens_encoding(tokenizer):\n    \"\"\"Ensure special tokens encode as single tokens in transformers v5.\n\n    Some model tokenizers (e.g. MiniCPM-V-4) define special tokens like <image>,\n    <slice> as attributes on the tokenizer class with corresponding IDs in the","sourceCodeStart":538,"sourceCodeEnd":574,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/utils/hf_transformers/tokenizer.py#L538-L574","documentation":"When --tokenizer-backend=fastokens is set, get_tokenizer routes loading through the fastokens library. If that loader raises any exception, SGLang wraps it in a RuntimeError pointing at fastokens and suggesting to drop the flag. The original exception is chained as the cause.","triggerScenarios":"Starting the server (or calling get_tokenizer/get_processor) with tokenizer_backend=\"fastokens\" for a model whose tokenizer files (tokenizer.json, sentencepiece, etc.) fastokens cannot parse.","commonSituations":"Enabling an experimental/alternative tokenizer backend on an uncommon model, a model with a non-standard tokenizer.json, or a model repo with only slow tokenizer files. Also transient HF download failures surface as this error.","solutions":["Re-run without --tokenizer-backend=fastokens to use the default HF backend.","Inspect the chained exception (`__cause__`) to see whether it is a real unsupported-tokenizer error or a download/permission issue.","Check the fastokens project (https://github.com/crusoecloud/fastokens) for supported tokenizer formats and update fastokens if it is outdated."],"exampleFix":"# before\npython -m sglang.launch_server --model ... --tokenizer-backend fastokens\n\n# after\npython -m sglang.launch_server --model ...  # default tokenizer backend","handlingStrategy":"fallback","validationCode":"try:\n    tok = get_tokenizer(name, tokenizer_backend=\"fastokens\")\nexcept RuntimeError as e:\n    if \"fastokens failed\" in str(e):\n        tok = get_tokenizer(name)  # default backend\n    else:\n        raise","typeGuard":null,"tryCatchPattern":"except RuntimeError as e: log warning, retry once with default tokenizer_backend; preserve e.__cause__ in logs.","preventionTips":["Smoke-test fastokens loading for a new model before enabling the flag in production.","Keep the original exception (__cause__) in logs for diagnosis."],"tags":["tokenizer","sglang","backend"],"backgroundTag":"unsupported-tokenizer-format","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}