sgl-project/sglang · warning

Using a slow tokenizer. This might cause a significant slowd

Error message

Using a slow tokenizer. This might cause a significant slowdown. Consider using a fast tokenizer instead.

What it means

After loading a tokenizer, SGLang checks whether the resulting object is a transformers PreTrainedTokenizerFast. If it is a slow (pure-Python) tokenizer — usually because the repo has no tokenizer.json and only tokenizer.py / sentencepiece — it warns that per-batch tokenization will be markedly slower and suggests a fast (Rust tokenizers) variant.

Source

Thrown at python/sglang/srt/utils/hf_transformers/tokenizer.py:433

    """Fix https://github.com/huggingface/transformers/pull/42563 which defaults
    special_tokens_pattern to "cls_sep", inserting None into token IDs when
    cls_token/sep_token are undefined (e.g. Kimi-VL's TikTokenTokenizer).
    """
    pattern = getattr(tokenizer, "special_tokens_pattern", None)
    if pattern == "cls_sep" and (
        tokenizer.cls_token_id is None or tokenizer.sep_token_id is None
    ):
        tokenizer.special_tokens_pattern = "none"


def _apply_post_load_fixes(tokenizer, tokenizer_name, revision):
    """Apply all post-load patches and return the final tokenizer."""
    _install_tokenizer_warnings_filter(tokenizer)
    _fix_v5_tokenizer_components(tokenizer, tokenizer_name, revision)
    _fix_v5_add_bos_eos_token(tokenizer, tokenizer_name, revision)

    if not isinstance(tokenizer, PreTrainedTokenizerFast):
        warnings.warn(
            "Using a slow tokenizer. This might cause a significant "
            "slowdown. Consider using a fast tokenizer instead."
        )

    patch_mistral_common_tokenizer(tokenizer)
    _fix_special_tokens_pattern(tokenizer)
    attach_additional_stop_token_ids(tokenizer)
    return patch_tokenizer(tokenizer)


# ---------------------------------------------------------------------------
# Public entry point
# ---------------------------------------------------------------------------


_fastokens_patched = False

View on GitHub (pinned to 0132848349)

Solutions

1. Convert the tokenizer to fast: transformers-cli convert slow->fast or AutoTokenizer.from_pretrained(id, use_fast=True).save_pretrained(dir) and point --tokenizer-path at the saved dir
2. Ensure tokenizer.json exists in the model repo / local dir and is not corrupted (delete and re-download)
3. For models where slow tokenization is required for correctness (rare, e.g. some Mistral installs), accept the warning and size capacity accordingly
4. Update transformers — newer versions auto-convert many slow tokenizers to fast

Example fix

# before: model dir has only tokenizer.model / tokenizer.py
# after: generate fast tokenizer once
from transformers import AutoTokenizer
AutoTokenizer.from_pretrained("model/id", use_fast=True).save_pretrained("./model-fast")
# then launch: --tokenizer-path ./model-fast

from transformers import AutoTokenizer, PreTrainedTokenizerFast
tok = AutoTokenizer.from_pretrained(model_id, use_fast=True)
assert isinstance(tok, PreTrainedTokenizerFast), "no fast tokenizer available for this repo"

from transformers import PreTrainedTokenizerFast
is_fast = isinstance(tokenizer, PreTrainedTokenizerFast)

When it happens

Trigger: get_tokenizer() loads a model whose HF repo lacks tokenizer.json (or conversion failed), yielding PreTrainedTokenizer instead of PreTrainedTokenizerFast; warning fires from _apply_post_load_fixes on every server/tokenizer startup with such a model.

Common situations: Older or niche models (early Llama variants, some multilingual/custom checkpoints) shipped only slow tokenizers; mistral-common patched tokenizers; local paths missing the fast tokenizer file.

Related errors

• think_end_token '{reasoning_parser.detector.think_end_token}
• Failed to load the tokenizer. If you are using a LLaMA V1 mo
• Failed to load the tokenizer. If the tokenizer is a custom t
• Failed to get server info. {error_data['error']['message']}
• world_size ({world_size}) is less than tensor_parallel_degre

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/8674a65442b5f20a. Report an issue: GitHub.