{"record":{"id":"8674a65442b5f20a","repo":"sgl-project/sglang","slug":"using-a-slow-tokenizer-this-might-cause-a-signifi","errorCode":null,"errorMessage":"Using a slow tokenizer. This might cause a significant slowdown. Consider using a fast tokenizer instead.","messagePattern":"Using a slow tokenizer\\. This might cause a significant slowdown\\. Consider using a fast tokenizer instead\\.","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"python/sglang/srt/utils/hf_transformers/tokenizer.py","lineNumber":433,"sourceCode":"    \"\"\"Fix https://github.com/huggingface/transformers/pull/42563 which defaults\n    special_tokens_pattern to \"cls_sep\", inserting None into token IDs when\n    cls_token/sep_token are undefined (e.g. Kimi-VL's TikTokenTokenizer).\n    \"\"\"\n    pattern = getattr(tokenizer, \"special_tokens_pattern\", None)\n    if pattern == \"cls_sep\" and (\n        tokenizer.cls_token_id is None or tokenizer.sep_token_id is None\n    ):\n        tokenizer.special_tokens_pattern = \"none\"\n\n\ndef _apply_post_load_fixes(tokenizer, tokenizer_name, revision):\n    \"\"\"Apply all post-load patches and return the final tokenizer.\"\"\"\n    _install_tokenizer_warnings_filter(tokenizer)\n    _fix_v5_tokenizer_components(tokenizer, tokenizer_name, revision)\n    _fix_v5_add_bos_eos_token(tokenizer, tokenizer_name, revision)\n\n    if not isinstance(tokenizer, PreTrainedTokenizerFast):\n        warnings.warn(\n            \"Using a slow tokenizer. This might cause a significant \"\n            \"slowdown. Consider using a fast tokenizer instead.\"\n        )\n\n    patch_mistral_common_tokenizer(tokenizer)\n    _fix_special_tokens_pattern(tokenizer)\n    attach_additional_stop_token_ids(tokenizer)\n    return patch_tokenizer(tokenizer)\n\n\n# ---------------------------------------------------------------------------\n# Public entry point\n# ---------------------------------------------------------------------------\n\n\n_fastokens_patched = False\n\n","sourceCodeStart":415,"sourceCodeEnd":451,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/utils/hf_transformers/tokenizer.py#L415-L451","documentation":"After loading a tokenizer, SGLang checks whether the resulting object is a transformers PreTrainedTokenizerFast. If it is a slow (pure-Python) tokenizer — usually because the repo has no tokenizer.json and only tokenizer.py / sentencepiece — it warns that per-batch tokenization will be markedly slower and suggests a fast (Rust tokenizers) variant.","triggerScenarios":"get_tokenizer() loads a model whose HF repo lacks tokenizer.json (or conversion failed), yielding PreTrainedTokenizer instead of PreTrainedTokenizerFast; warning fires from _apply_post_load_fixes on every server/tokenizer startup with such a model.","commonSituations":"Older or niche models (early Llama variants, some multilingual/custom checkpoints) shipped only slow tokenizers; mistral-common patched tokenizers; local paths missing the fast tokenizer file.","solutions":["Convert the tokenizer to fast: transformers-cli convert slow->fast or AutoTokenizer.from_pretrained(id, use_fast=True).save_pretrained(dir) and point --tokenizer-path at the saved dir","Ensure tokenizer.json exists in the model repo / local dir and is not corrupted (delete and re-download)","For models where slow tokenization is required for correctness (rare, e.g. some Mistral installs), accept the warning and size capacity accordingly","Update transformers — newer versions auto-convert many slow tokenizers to fast"],"exampleFix":"# before: model dir has only tokenizer.model / tokenizer.py\n# after: generate fast tokenizer once\nfrom transformers import AutoTokenizer\nAutoTokenizer.from_pretrained(\"model/id\", use_fast=True).save_pretrained(\"./model-fast\")\n# then launch: --tokenizer-path ./model-fast","handlingStrategy":"type-guard","validationCode":"from transformers import AutoTokenizer, PreTrainedTokenizerFast\ntok = AutoTokenizer.from_pretrained(model_id, use_fast=True)\nassert isinstance(tok, PreTrainedTokenizerFast), \"no fast tokenizer available for this repo\"","typeGuard":"from transformers import PreTrainedTokenizerFast\nis_fast = isinstance(tokenizer, PreTrainedTokenizerFast)","tryCatchPattern":null,"preventionTips":["Pre-convert tokenizers to fast and cache them with the model","Verify tokenizer.json exists in the repo before deploying","Budget capacity assuming slow tokenization if fast is unavailable"],"tags":["tokenizer","performance","huggingface","startup"],"backgroundTag":"slow-tokenizer-warning","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}