sgl-project/sglang · error · ValueError

No pre-tokenizer regex known for tokenizer.ggml.pre={pre_nam

Error message

No pre-tokenizer regex known for tokenizer.ggml.pre={pre_name!r}; known: {sorted(_PRE_TOKENIZER_REGEX)}

What it means

When building a tokenizer natively from GGUF metadata, the tokenizer.ggml.pre value is not in sglang's _PRE_TOKENIZER_REGEX table, so no pre-tokenizer regex can be selected and tokenization would be wrong. The error lists supported pre-tokenizer names.

Source

Thrown at python/sglang/srt/utils/hf_transformers/gguf_native.py:149

    over the NORMAL tokens, the CONTROL tokens registered as added specials, and
    the split regex named by ``tokenizer.ggml.pre``.
    """
    import json

    from gguf import GGUFReader
    from tokenizers import Tokenizer
    from transformers import PreTrainedTokenizerFast

    reader = GGUFReader(gguf_path)
    meta = {key: field.contents() for key, field in reader.fields.items()}

    tokens = list(meta["tokenizer.ggml.tokens"])
    token_types = [int(t) for t in meta["tokenizer.ggml.token_type"]]
    merges = [tuple(m.split(" ", 1)) for m in meta["tokenizer.ggml.merges"]]

    pre_name = meta.get("tokenizer.ggml.pre")
    if pre_name not in _PRE_TOKENIZER_REGEX:
        raise ValueError(
            f"No pre-tokenizer regex known for tokenizer.ggml.pre={pre_name!r}; "
            f"known: {sorted(_PRE_TOKENIZER_REGEX)}"
        )

    control_ids = [
        i for i, t in enumerate(token_types) if t == _GGML_TOKEN_TYPE_CONTROL
    ]
    control = set(control_ids)
    vocab = {tok: i for i, tok in enumerate(tokens) if i not in control}

    def token_of(key):
        idx = meta.get(f"tokenizer.ggml.{key}")
        return None if idx is None else tokens[int(idx)]

    bos = token_of("bos_token_id")

    spec = {
        "version": "1.0",

View on GitHub (pinned to 0132848349)

Solutions

Upgrade sglang so _PRE_TOKENIZER_REGEX includes the new pre type
Re-convert/re-download the GGUF with a standard pre-tokenizer setting
Or serve the model from its HF (non-GGUF) weights, which use tokenizer.json directly

Defensive patterns

Strategy: try-catch

Validate before calling

pre = gguf_metadata.get('tokenizer.ggml.pre')
if pre not in KNOWN_PRE_TOKENIZERS: warn early and fall back to HF weights

Try / catch

try:
    get_tokenizer(model)
except ValueError as e:
    if 'pre-tokenizer regex' in str(e): serve HF-format weights instead

Prevention

Prefer official GGUF conversions that bundle tokenizer.json
Pin sglang versions compatible with your GGUF generation

When it happens

Trigger: Loading a GGUF whose tokenizer was converted with a new or unusual pre-tokenizer (e.g. a newly added BPE variant like 'jinja' or 'gpt-4o') not yet known to this sglang version.

Common situations: Freshly converted GGUFs from llama.cpp supporting newer chat models while the sglang install lags behind.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/1d9a33020ef878a4. Report an issue: GitHub.