Comfy-Org/ComfyUI · error · ValueError

Unsupported tokenizer split pattern: {pattern_str}

Error message

Unsupported tokenizer split pattern: {pattern_str}

What it means

comfy/text_encoders/bpe_tokenizer.py implements a BPE tokenizer that supports exactly one pre-tokenization regex: the Llama split pattern (_LLAMA_PATTERN, the standard GPT-style \p{L}/\p{N} pattern). _make_split_pattern compares the pattern string from tokenizer.json byte-for-byte against that one pattern and raises for anything else, including trivially equivalent variants with different whitespace or escaping.

Source

Thrown at comfy/text_encoders/bpe_tokenizer.py:98

            while j < len(text) and _is_whitespace(text[j]):
                j += 1
            last_newline = max(text.rfind("\r", i, j), text.rfind("\n", i, j))
            if last_newline >= i:
                j = last_newline + 1
            elif j < len(text) and j - i > 1:
                j -= 1
            pieces.append(text[i:j])
            i = j
            continue

        pieces.append(text[i])
        i += 1
    return pieces


def _make_split_pattern(pattern_str):
    if pattern_str != _LLAMA_PATTERN:
        raise ValueError(f"Unsupported tokenizer split pattern: {pattern_str}")
    return _split_llama


def _bytes_to_unicode():
    bs = (list(range(ord("!"), ord("~") + 1))
          + list(range(ord("¡"), ord("¬") + 1))
          + list(range(ord("®"), ord("ÿ") + 1)))
    cs = bs[:]
    n = 0
    for b in range(2**8):
        if b not in bs:
            bs.append(b)
            cs.append(2**8 + n)
            n += 1
    cs = [chr(n) for n in cs]
    return dict(zip(bs, cs))

View on GitHub (pinned to 1c6d8d45b3)

Solutions

1. Use a tokenizer.json whose pre_tokenizer pattern is the standard Llama pattern; tokenizers from Llama-lineage models (and most Llama-compatible fine-tunes) match.
2. Open tokenizer.json and inspect data['pre_tokenizer'] to see the offending pattern string.
3. If the model genuinely needs a different split pattern, its text encoder must be added to ComfyUI proper rather than forced through this BPE loader.

from comfy.text_encoders.bpe_tokenizer import _LLAMA_PATTERN
pattern = tokenizer_json['pre_tokenizer']['pattern']['Pattern'] if tokenizer_json.get('pre_tokenizer') else None
assert pattern == _LLAMA_PATTERN, 'tokenizer.json uses a non-Llama split pattern'

try:
    tok = from_tokenizer_json(data)
except ValueError as e:
    if 'Unsupported tokenizer split pattern' in str(e):
        raise SystemExit('Use a Llama-pattern tokenizer.json for this encoder.')
    raise

When it happens

Trigger: Loading a tokenizer.json whose pre_tokenizer regex is a different model family's pattern (GPT-2, Mistral tekken, Gemma, Phi); a pattern string that differs only in regex escaping or whitespace from _LLAMA_PATTERN; converting tokenizers from other ecosystems that embed their own split regex.

Common situations: Pointing a text-encoder loader at a tokenizer.json from an unsupported model; tokenizer.json files regenerated by a different tokenizer library version that re-serializes the regex; hand-edited tokenizer configs.

Related errors

• Could not extract regex pattern from {tok_file}
• Unknown patch method: {self.patch_method}
• Image-token count {image_idx.shape[0]} != ViT output count {
• MiniMax Music3 prompt has {prompt_tokens} tokens; maximum is
• MiniMax Music3 tokenizer mismatch for {token}: expected {exp

AI-assisted analysis of Comfy-Org/ComfyUI@1c6d8d45b3 (2026-08-14). Data as JSON: /api/errors/fd44484776189aaa. Report an issue: GitHub.