sgl-project/sglang · error · RuntimeError

fastokens failed to load tokenizer for {tokenizer_name!r}. T

Error message

fastokens failed to load tokenizer for {tokenizer_name!r}. This model's tokenizer may not be supported by fastokens — see https://github.com/crusoecloud/fastokens. Re-run without --tokenizer-backend=fastokens to use the default backend.

What it means

When --tokenizer-backend=fastokens is set, get_tokenizer routes loading through the fastokens library. If that loader raises any exception, SGLang wraps it in a RuntimeError pointing at fastokens and suggesting to drop the flag. The original exception is chained as the cause.

Source

Thrown at python/sglang/srt/utils/hf_transformers/tokenizer.py:556

            tokenizer = _auto_tokenizer_from_pretrained(
                tokenizer_name, *args, **common_kwargs
            )

            # With fastokens, the patched TokenizersBackend.from_pretrained already
            # returned a tokenizer whose backend is a fastokens shim. Re-resolving via
            # the declared class (e.g. Qwen2Tokenizer) would discard that work.
            if (
                type(tokenizer).__name__ == _TOKENIZERS_BACKEND
                and tokenizer_backend != "fastokens"
            ):
                tokenizer = _resolve_tokenizers_backend(
                    tokenizer_name, *args, **common_kwargs
                )

        return _apply_post_load_fixes(tokenizer, tokenizer_name, tokenizer_revision)
    except Exception as e:
        if tokenizer_backend == "fastokens":
            raise RuntimeError(
                f"fastokens failed to load tokenizer for {tokenizer_name!r}. "
                f"This model's tokenizer may not be supported by fastokens — "
                f"see https://github.com/crusoecloud/fastokens. "
                f"Re-run without --tokenizer-backend=fastokens to use the default backend."
            ) from e
        raise


# ---------------------------------------------------------------------------
# Exported helpers (used by processor.py, etc.)
# ---------------------------------------------------------------------------


def _fix_added_tokens_encoding(tokenizer):
    """Ensure special tokens encode as single tokens in transformers v5.

    Some model tokenizers (e.g. MiniCPM-V-4) define special tokens like <image>,
    <slice> as attributes on the tokenizer class with corresponding IDs in the

View on GitHub (pinned to 0132848349)

Solutions

Re-run without --tokenizer-backend=fastokens to use the default HF backend.
Inspect the chained exception (`__cause__`) to see whether it is a real unsupported-tokenizer error or a download/permission issue.
Check the fastokens project (https://github.com/crusoecloud/fastokens) for supported tokenizer formats and update fastokens if it is outdated.

Example fix

# before
python -m sglang.launch_server --model ... --tokenizer-backend fastokens

# after
python -m sglang.launch_server --model ...  # default tokenizer backend

Defensive patterns

Strategy: fallback

Validate before calling

try:
    tok = get_tokenizer(name, tokenizer_backend="fastokens")
except RuntimeError as e:
    if "fastokens failed" in str(e):
        tok = get_tokenizer(name)  # default backend
    else:
        raise

Try / catch

except RuntimeError as e: log warning, retry once with default tokenizer_backend; preserve e.__cause__ in logs.

Prevention

Smoke-test fastokens loading for a new model before enabling the flag in production.
Keep the original exception (__cause__) in logs for diagnosis.

When it happens

Trigger: Starting the server (or calling get_tokenizer/get_processor) with tokenizer_backend="fastokens" for a model whose tokenizer files (tokenizer.json, sentencepiece, etc.) fastokens cannot parse.

Common situations: Enabling an experimental/alternative tokenizer backend on an uncommon model, a model with a non-standard tokenizer.json, or a model repo with only slow tokenizer files. Also transient HF download failures surface as this error.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/5c0d1a748a9260b8. Report an issue: GitHub.