{"record":{"id":"fd44484776189aaa","repo":"Comfy-Org/ComfyUI","slug":"unsupported-tokenizer-split-pattern-pattern-str","errorCode":null,"errorMessage":"Unsupported tokenizer split pattern: {pattern_str}","messagePattern":"Unsupported tokenizer split pattern: (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"comfy/text_encoders/bpe_tokenizer.py","lineNumber":98,"sourceCode":"            while j < len(text) and _is_whitespace(text[j]):\n                j += 1\n            last_newline = max(text.rfind(\"\\r\", i, j), text.rfind(\"\\n\", i, j))\n            if last_newline >= i:\n                j = last_newline + 1\n            elif j < len(text) and j - i > 1:\n                j -= 1\n            pieces.append(text[i:j])\n            i = j\n            continue\n\n        pieces.append(text[i])\n        i += 1\n    return pieces\n\n\ndef _make_split_pattern(pattern_str):\n    if pattern_str != _LLAMA_PATTERN:\n        raise ValueError(f\"Unsupported tokenizer split pattern: {pattern_str}\")\n    return _split_llama\n\n\ndef _bytes_to_unicode():\n    bs = (list(range(ord(\"!\"), ord(\"~\") + 1))\n          + list(range(ord(\"¡\"), ord(\"¬\") + 1))\n          + list(range(ord(\"®\"), ord(\"ÿ\") + 1)))\n    cs = bs[:]\n    n = 0\n    for b in range(2**8):\n        if b not in bs:\n            bs.append(b)\n            cs.append(2**8 + n)\n            n += 1\n    cs = [chr(n) for n in cs]\n    return dict(zip(bs, cs))\n\n","sourceCodeStart":80,"sourceCodeEnd":116,"githubUrl":"https://github.com/Comfy-Org/ComfyUI/blob/1c6d8d45b3693bfbb32385b410d813a7fd6be216/comfy/text_encoders/bpe_tokenizer.py#L80-L116","documentation":"comfy/text_encoders/bpe_tokenizer.py implements a BPE tokenizer that supports exactly one pre-tokenization regex: the Llama split pattern (_LLAMA_PATTERN, the standard GPT-style \\p{L}/\\p{N} pattern). _make_split_pattern compares the pattern string from tokenizer.json byte-for-byte against that one pattern and raises for anything else, including trivially equivalent variants with different whitespace or escaping.","triggerScenarios":"Loading a tokenizer.json whose pre_tokenizer regex is a different model family's pattern (GPT-2, Mistral tekken, Gemma, Phi); a pattern string that differs only in regex escaping or whitespace from _LLAMA_PATTERN; converting tokenizers from other ecosystems that embed their own split regex.","commonSituations":"Pointing a text-encoder loader at a tokenizer.json from an unsupported model; tokenizer.json files regenerated by a different tokenizer library version that re-serializes the regex; hand-edited tokenizer configs.","solutions":["Use a tokenizer.json whose pre_tokenizer pattern is the standard Llama pattern; tokenizers from Llama-lineage models (and most Llama-compatible fine-tunes) match.","Open tokenizer.json and inspect data['pre_tokenizer'] to see the offending pattern string.","If the model genuinely needs a different split pattern, its text encoder must be added to ComfyUI proper rather than forced through this BPE loader."],"exampleFix":null,"handlingStrategy":"validation","validationCode":"from comfy.text_encoders.bpe_tokenizer import _LLAMA_PATTERN\npattern = tokenizer_json['pre_tokenizer']['pattern']['Pattern'] if tokenizer_json.get('pre_tokenizer') else None\nassert pattern == _LLAMA_PATTERN, 'tokenizer.json uses a non-Llama split pattern'","typeGuard":null,"tryCatchPattern":"try:\n    tok = from_tokenizer_json(data)\nexcept ValueError as e:\n    if 'Unsupported tokenizer split pattern' in str(e):\n        raise SystemExit('Use a Llama-pattern tokenizer.json for this encoder.')\n    raise","preventionTips":["Ship the tokenizer.json that ships with the supported model family.","Do not re-serialize tokenizer.json with other tools that rewrite the regex."],"tags":["tokenizer","bpe","llama","regex","unsupported-format"],"backgroundTag":null,"analyzedSha":"1c6d8d45b3693bfbb32385b410d813a7fd6be216","analyzedAt":"2026-08-14T19:37:18.893Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}