{"record":{"id":"b14a075a4d5313e5","repo":"sgl-project/sglang","slug":"retry-with-use-fast-false-for-tokenizer-name-als","errorCode":null,"errorMessage":"Retry with use_fast=False for {tokenizer_name} also failed (initial load returned TokenizersBackend): {e}","messagePattern":"Retry with use_fast=False for (.+?) also failed \\(initial load returned TokenizersBackend\\): (.+?)","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/utils/hf_transformers/tokenizer.py","lineNumber":232,"sourceCode":"\n    In transformers v5, ``AutoTokenizer`` falls back to ``TokenizersBackend``\n    when the model_type has no tokenizer mapping.  This retries with\n    ``use_fast=False``, then attempts loading by the class declared in\n    ``tokenizer_config.json``.  May still return a ``TokenizersBackend``\n    if all retries fail (with a warning).\n    \"\"\"\n    logger.debug(\n        \"Tokenizer loaded as generic TokenizersBackend for %s, \"\n        \"retrying with use_fast=False\",\n        tokenizer_name,\n    )\n    common_kwargs = {**common_kwargs, \"use_fast\": False}\n    try:\n        tokenizer = AutoTokenizer.from_pretrained(\n            tokenizer_name, *args, **common_kwargs\n        )\n    except (ValueError, TypeError, OSError, ImportError, RuntimeError) as e:\n        raise RuntimeError(\n            f\"Retry with use_fast=False for {tokenizer_name} also failed \"\n            f\"(initial load returned TokenizersBackend): {e}\"\n        ) from e\n\n    if type(tokenizer).__name__ == _TOKENIZERS_BACKEND:\n        tokenizer = (\n            _load_tokenizer_by_declared_class(tokenizer_name, *args, **common_kwargs)\n            or tokenizer\n        )\n\n    if type(tokenizer).__name__ == _TOKENIZERS_BACKEND:\n        if common_kwargs.get(\"trust_remote_code\"):\n            logger.warning(\n                \"Tokenizer for %s is still TokenizersBackend after retries \"\n                \"with --trust-remote-code. Model-specific tokenizer attributes \"\n                \"may be missing.\",\n                tokenizer_name,\n            )","sourceCodeStart":214,"sourceCodeEnd":250,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/utils/hf_transformers/tokenizer.py#L214-L250","documentation":"When the first AutoTokenizer load returned a generic TokenizersBackend instead of a concrete class, sglang retries with use_fast=False; that retry also raised, and this RuntimeError wraps the retry failure with its cause.","triggerScenarios":"Loading a tokenizer whose fast path yields TokenizersBackend and whose slow (Python) path is also broken or unavailable for the installed transformers version.","commonSituations":"New tokenizers on transformers versions where the backend shim is incomplete, or missing slow tokenizer implementations.","solutions":["Upgrade (or pin) transformers to a version matching the model's tokenizer support","Prefer a repo with a proper tokenizer.json plus concrete tokenizer class","Inspect the chained exception e for the real cause (missing file, unsupported type, etc.)"],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"try:\n    get_tokenizer(model)\nexcept RuntimeError as e:\n    if 'use_fast=False' in str(e): pin/upgrade transformers to the model's recommended version, then retry","preventionTips":["Pin a transformers version validated for your model","Inspect chained cause e.__cause__ before guessing"],"tags":["tokenizer","transformers","version-mismatch"],"backgroundTag":"tokenizer-load-failed","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}