Comfy-Org/ComfyUI · error · ValueError

Could not extract regex pattern from {tok_file}

Error message

Could not extract regex pattern from {tok_file}

What it means

In from_tokenizer_json (comfy/text_encoders/bpe_tokenizer.py), _extract_pattern parses the pre_tokenizer section of tokenizer.json and returns None for any structure it cannot map to the supported Llama pattern. A None result aborts tokenizer construction with this ValueError naming the file, because tokenization would be silently wrong without a known split rule.

Source

Thrown at comfy/text_encoders/bpe_tokenizer.py:249

    with open(tok_file, encoding='utf-8') as f:
        data = json.load(f)

    vocab = dict(data['model']['vocab'])  # str -> int

    merges_by_pair = {}
    for i, merge_str in enumerate(data['model'].get('merges', [])):
        a, b = merge_str.split(' ', 1)
        if (a, b) not in merges_by_pair:
            merges_by_pair[(a, b)] = i

    special_token_ids = {}
    for tok in data.get('added_tokens', []):
        special_token_ids[tok['content']] = tok['id']
        vocab[tok['content']] = tok['id']  # include in vocab for inv_vocab decode

    pattern = _extract_pattern(data.get('pre_tokenizer', {}))
    if pattern is None:
        raise ValueError(f"Could not extract regex pattern from {tok_file}")

    bos_id = _extract_bos_id(data.get('post_processor', {}), special_token_ids)

    byte_encoder = _bytes_to_unicode()
    byte_decoder = {v: k for k, v in byte_encoder.items()}

    return BPETokenizer(vocab, merges_by_pair, special_token_ids, pattern,
                        byte_encoder, byte_decoder, bos_id=bos_id)


def from_tekken_json(data):
    """Build a BPETokenizer from a Mistral tekken JSON blob (bytes or str)."""
    mistral_vocab = json.loads(data)
    config = mistral_vocab["config"]

    byte_encoder = _bytes_to_unicode()
    byte_decoder = {v: k for k, v in byte_encoder.items()}

View on GitHub (pinned to 1c6d8d45b3)

Solutions

1. Inspect the 'pre_tokenizer' field of the failing tokenizer.json to see which shape it uses.
2. Supply a tokenizer.json from a Llama-lineage model whose pre_tokenizer matches the supported pattern exactly.
3. Update ComfyUI if a newer release added support for that pre_tokenizer serialization.
4. Re-export the tokenizer with the `tokenizers` library producing the single-regex pre_tokenizer form.

pre = tokenizer_json.get('pre_tokenizer') or {}
assert pre.get('type') == 'Split' and pre.get('pattern', {}).get('Regex'), 'unsupported pre_tokenizer structure'

try:
    tok = from_tokenizer_json(data)
except ValueError as e:
    if 'Could not extract regex pattern' in str(e):
        raise SystemExit('tokenizer.json pre_tokenizer is missing or in an unsupported shape.')
    raise

When it happens

Trigger: tokenizer.json with a missing pre_tokenizer section, a pre_tokenizer list/sequence wrapper, a different regex than _LLAMA_PATTERN, or a different tokenizer 'type' field; tokenizers saved by newer versions of the `tokenizers` library using a different serialization shape.

Common situations: Downloading tokenizer.json from a non-Llama model (GPT-2, Qwen, tekken); tokenizer.json produced by tokenizers>=0.20 sequence-style pre_tokenizers; files where the regex was re-escaped (\\p{L} vs \p{L}) on re-save.

Related errors

• Unsupported tokenizer split pattern: {pattern_str}
• Unknown patch method: {self.patch_method}
• Image-token count {image_idx.shape[0]} != ViT output count {
• MiniMax Music3 prompt has {prompt_tokens} tokens; maximum is
• MiniMax Music3 tokenizer mismatch for {token}: expected {exp

AI-assisted analysis of Comfy-Org/ComfyUI@1c6d8d45b3 (2026-08-14). Data as JSON: /api/errors/143d8f302b13d336. Report an issue: GitHub.