{"record":{"id":"1d9a33020ef878a4","repo":"sgl-project/sglang","slug":"no-pre-tokenizer-regex-known-for-tokenizer-ggml-pr","errorCode":null,"errorMessage":"No pre-tokenizer regex known for tokenizer.ggml.pre={pre_name!r}; known: {sorted(_PRE_TOKENIZER_REGEX)}","messagePattern":"No pre-tokenizer regex known for tokenizer\\.ggml\\.pre=(.+?); known: (.+?)","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/utils/hf_transformers/gguf_native.py","lineNumber":149,"sourceCode":"    over the NORMAL tokens, the CONTROL tokens registered as added specials, and\n    the split regex named by ``tokenizer.ggml.pre``.\n    \"\"\"\n    import json\n\n    from gguf import GGUFReader\n    from tokenizers import Tokenizer\n    from transformers import PreTrainedTokenizerFast\n\n    reader = GGUFReader(gguf_path)\n    meta = {key: field.contents() for key, field in reader.fields.items()}\n\n    tokens = list(meta[\"tokenizer.ggml.tokens\"])\n    token_types = [int(t) for t in meta[\"tokenizer.ggml.token_type\"]]\n    merges = [tuple(m.split(\" \", 1)) for m in meta[\"tokenizer.ggml.merges\"]]\n\n    pre_name = meta.get(\"tokenizer.ggml.pre\")\n    if pre_name not in _PRE_TOKENIZER_REGEX:\n        raise ValueError(\n            f\"No pre-tokenizer regex known for tokenizer.ggml.pre={pre_name!r}; \"\n            f\"known: {sorted(_PRE_TOKENIZER_REGEX)}\"\n        )\n\n    control_ids = [\n        i for i, t in enumerate(token_types) if t == _GGML_TOKEN_TYPE_CONTROL\n    ]\n    control = set(control_ids)\n    vocab = {tok: i for i, tok in enumerate(tokens) if i not in control}\n\n    def token_of(key):\n        idx = meta.get(f\"tokenizer.ggml.{key}\")\n        return None if idx is None else tokens[int(idx)]\n\n    bos = token_of(\"bos_token_id\")\n\n    spec = {\n        \"version\": \"1.0\",","sourceCodeStart":131,"sourceCodeEnd":167,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/utils/hf_transformers/gguf_native.py#L131-L167","documentation":"When building a tokenizer natively from GGUF metadata, the tokenizer.ggml.pre value is not in sglang's _PRE_TOKENIZER_REGEX table, so no pre-tokenizer regex can be selected and tokenization would be wrong. The error lists supported pre-tokenizer names.","triggerScenarios":"Loading a GGUF whose tokenizer was converted with a new or unusual pre-tokenizer (e.g. a newly added BPE variant like 'jinja' or 'gpt-4o') not yet known to this sglang version.","commonSituations":"Freshly converted GGUFs from llama.cpp supporting newer chat models while the sglang install lags behind.","solutions":["Upgrade sglang so _PRE_TOKENIZER_REGEX includes the new pre type","Re-convert/re-download the GGUF with a standard pre-tokenizer setting","Or serve the model from its HF (non-GGUF) weights, which use tokenizer.json directly"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"pre = gguf_metadata.get('tokenizer.ggml.pre')\nif pre not in KNOWN_PRE_TOKENIZERS: warn early and fall back to HF weights","typeGuard":null,"tryCatchPattern":"try:\n    get_tokenizer(model)\nexcept ValueError as e:\n    if 'pre-tokenizer regex' in str(e): serve HF-format weights instead","preventionTips":["Prefer official GGUF conversions that bundle tokenizer.json","Pin sglang versions compatible with your GGUF generation"],"tags":["gguf","tokenizer","version-mismatch"],"backgroundTag":"unsupported-tokenizer-format","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}