{"record":{"id":"143d8f302b13d336","repo":"Comfy-Org/ComfyUI","slug":"could-not-extract-regex-pattern-from-tok-file","errorCode":null,"errorMessage":"Could not extract regex pattern from {tok_file}","messagePattern":"Could not extract regex pattern from (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"comfy/text_encoders/bpe_tokenizer.py","lineNumber":249,"sourceCode":"    with open(tok_file, encoding='utf-8') as f:\n        data = json.load(f)\n\n    vocab = dict(data['model']['vocab'])  # str -> int\n\n    merges_by_pair = {}\n    for i, merge_str in enumerate(data['model'].get('merges', [])):\n        a, b = merge_str.split(' ', 1)\n        if (a, b) not in merges_by_pair:\n            merges_by_pair[(a, b)] = i\n\n    special_token_ids = {}\n    for tok in data.get('added_tokens', []):\n        special_token_ids[tok['content']] = tok['id']\n        vocab[tok['content']] = tok['id']  # include in vocab for inv_vocab decode\n\n    pattern = _extract_pattern(data.get('pre_tokenizer', {}))\n    if pattern is None:\n        raise ValueError(f\"Could not extract regex pattern from {tok_file}\")\n\n    bos_id = _extract_bos_id(data.get('post_processor', {}), special_token_ids)\n\n    byte_encoder = _bytes_to_unicode()\n    byte_decoder = {v: k for k, v in byte_encoder.items()}\n\n    return BPETokenizer(vocab, merges_by_pair, special_token_ids, pattern,\n                        byte_encoder, byte_decoder, bos_id=bos_id)\n\n\ndef from_tekken_json(data):\n    \"\"\"Build a BPETokenizer from a Mistral tekken JSON blob (bytes or str).\"\"\"\n    mistral_vocab = json.loads(data)\n    config = mistral_vocab[\"config\"]\n\n    byte_encoder = _bytes_to_unicode()\n    byte_decoder = {v: k for k, v in byte_encoder.items()}\n","sourceCodeStart":231,"sourceCodeEnd":267,"githubUrl":"https://github.com/Comfy-Org/ComfyUI/blob/1c6d8d45b3693bfbb32385b410d813a7fd6be216/comfy/text_encoders/bpe_tokenizer.py#L231-L267","documentation":"In from_tokenizer_json (comfy/text_encoders/bpe_tokenizer.py), _extract_pattern parses the pre_tokenizer section of tokenizer.json and returns None for any structure it cannot map to the supported Llama pattern. A None result aborts tokenizer construction with this ValueError naming the file, because tokenization would be silently wrong without a known split rule.","triggerScenarios":"tokenizer.json with a missing pre_tokenizer section, a pre_tokenizer list/sequence wrapper, a different regex than _LLAMA_PATTERN, or a different tokenizer 'type' field; tokenizers saved by newer versions of the `tokenizers` library using a different serialization shape.","commonSituations":"Downloading tokenizer.json from a non-Llama model (GPT-2, Qwen, tekken); tokenizer.json produced by tokenizers>=0.20 sequence-style pre_tokenizers; files where the regex was re-escaped (\\\\p{L} vs \\p{L}) on re-save.","solutions":["Inspect the 'pre_tokenizer' field of the failing tokenizer.json to see which shape it uses.","Supply a tokenizer.json from a Llama-lineage model whose pre_tokenizer matches the supported pattern exactly.","Update ComfyUI if a newer release added support for that pre_tokenizer serialization.","Re-export the tokenizer with the `tokenizers` library producing the single-regex pre_tokenizer form."],"exampleFix":null,"handlingStrategy":"validation","validationCode":"pre = tokenizer_json.get('pre_tokenizer') or {}\nassert pre.get('type') == 'Split' and pre.get('pattern', {}).get('Regex'), 'unsupported pre_tokenizer structure'","typeGuard":null,"tryCatchPattern":"try:\n    tok = from_tokenizer_json(data)\nexcept ValueError as e:\n    if 'Could not extract regex pattern' in str(e):\n        raise SystemExit('tokenizer.json pre_tokenizer is missing or in an unsupported shape.')\n    raise","preventionTips":["Verify the pre_tokenizer section exists and is a single Split/Regex entry before loading.","Prefer tokenizers exported alongside ComfyUI-supported models."],"tags":["tokenizer","bpe","pre-tokenizer","json-config"],"backgroundTag":null,"analyzedSha":"1c6d8d45b3693bfbb32385b410d813a7fd6be216","analyzedAt":"2026-08-14T19:37:18.893Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}