{"record":{"id":"62aa5416e7f05465","repo":"sgl-project/sglang","slug":"the-fastokens-package-is-required-when-tokenizer","errorCode":null,"errorMessage":"The fastokens package is required when --tokenizer-backend=fastokens. Install it with: pip install 'sglang[fastokens]'","messagePattern":"The fastokens package is required when --tokenizer-backend=fastokens\\. Install it with: pip install 'sglang\\[fastokens\\]'","errorType":"exception","errorClass":"ImportError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/utils/hf_transformers/tokenizer.py","lineNumber":460,"sourceCode":"\n\n# ---------------------------------------------------------------------------\n# Public entry point\n# ---------------------------------------------------------------------------\n\n\n_fastokens_patched = False\n\n\ndef _ensure_fastokens_patched():\n    \"\"\"Monkey-patch transformers to use the fastokens backend (once).\"\"\"\n    global _fastokens_patched\n    if _fastokens_patched:\n        return\n    try:\n        import fastokens\n    except ImportError:\n        raise ImportError(\n            \"The fastokens package is required when --tokenizer-backend=fastokens. \"\n            \"Install it with: pip install 'sglang[fastokens]'\"\n        ) from None\n\n    fastokens.patch_transformers()\n    _fastokens_patched = True\n    logger.info(\"fastokens backend enabled - transformers patched successfully\")\n\n\ndef get_tokenizer(\n    tokenizer_name: str,\n    *args,\n    tokenizer_mode: str = \"auto\",\n    trust_remote_code: bool = False,\n    tokenizer_revision: Optional[str] = None,\n    tokenizer_backend: str = \"huggingface\",\n    **kwargs,\n) -> Union[PreTrainedTokenizer, PreTrainedTokenizerFast]:","sourceCodeStart":442,"sourceCodeEnd":478,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/utils/hf_transformers/tokenizer.py#L442-L478","documentation":"--tokenizer-backend=fastokens was requested (in get_tokenizer/get_processor), but the optional fastokens package is not installed, so it cannot patch transformers. Install it via the sglang[fastokens] extra.","triggerScenarios":"Passing --tokenizer-backend fastokens in an environment where 'import fastokens' fails.","commonSituations":"Using the flag after seeing it in docs/benchmarks without installing the optional dependency.","solutions":["pip install 'sglang[fastokens]'","Or drop --tokenizer-backend fastokens and use the default HF tokenizer backend"],"exampleFix":"# before: command fails with ImportError\n# after\npip install 'sglang[fastokens]'","handlingStrategy":"validation","validationCode":"if tokenizer_backend == 'fastokens':\n    import importlib.util\n    assert importlib.util.find_spec('fastokens') is not None, 'pip install sglang[fastokens]'","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Install the sglang[fastokens] extra before enabling the backend","Or omit the flag to use the default tokenizer backend"],"tags":["tokenizer","missing-dependency","installation"],"backgroundTag":"missing-optional-dependency","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}