{"record":{"id":"c82d4ce2ac4b2d74","repo":"huggingface/tokenizers","slug":"cls-token-not-found-in-the-vocabulary","errorCode":null,"errorMessage":"cls_token not found in the vocabulary","messagePattern":"cls_token not found in the vocabulary","errorType":"exception","errorClass":"TypeError","httpStatus":null,"severity":"error","filePath":"bindings/python/py_src/tokenizers/implementations/bert_wordpiece.py","lineNumber":60,"sourceCode":"            tokenizer.add_special_tokens([str(pad_token)])\n        if tokenizer.token_to_id(str(mask_token)) is not None:\n            tokenizer.add_special_tokens([str(mask_token)])\n\n        tokenizer.normalizer = BertNormalizer(\n            clean_text=clean_text,\n            handle_chinese_chars=handle_chinese_chars,\n            strip_accents=strip_accents,\n            lowercase=lowercase,\n        )\n        tokenizer.pre_tokenizer = BertPreTokenizer()\n\n        if vocab is not None:\n            sep_token_id = tokenizer.token_to_id(str(sep_token))\n            if sep_token_id is None:\n                raise TypeError(\"sep_token not found in the vocabulary\")\n            cls_token_id = tokenizer.token_to_id(str(cls_token))\n            if cls_token_id is None:\n                raise TypeError(\"cls_token not found in the vocabulary\")\n\n            tokenizer.post_processor = BertProcessing((str(sep_token), sep_token_id), (str(cls_token), cls_token_id))\n        tokenizer.decoder = decoders.WordPiece(prefix=wordpieces_prefix)\n\n        parameters = {\n            \"model\": \"BertWordPiece\",\n            \"unk_token\": unk_token,\n            \"sep_token\": sep_token,\n            \"cls_token\": cls_token,\n            \"pad_token\": pad_token,\n            \"mask_token\": mask_token,\n            \"clean_text\": clean_text,\n            \"handle_chinese_chars\": handle_chinese_chars,\n            \"strip_accents\": strip_accents,\n            \"lowercase\": lowercase,\n            \"wordpieces_prefix\": wordpieces_prefix,\n        }\n","sourceCodeStart":42,"sourceCodeEnd":78,"githubUrl":"https://github.com/huggingface/tokenizers/blob/6cfd9d385ca0ed91c10b49f0ce97d02cfde1b607/bindings/python/py_src/tokenizers/implementations/bert_wordpiece.py#L42-L78","documentation":"`BertWordPieceTokenizer.__init__` raises this `TypeError` when a vocab file is provided but the configured `cls_token` (default `\"[CLS]\"`) is not present in the vocabulary. The BERT post-processor needs both SEP and CLS token ids, so the constructor validates CLS right after SEP and refuses to build an inconsistent tokenizer.","triggerScenarios":"Constructing `BertWordPieceTokenizer(vocab_file, cls_token=\"[CLS]\")` where the vocab lacks that exact string — e.g. vocab uses lowercase `[cls]`, another marker like `<s>`, or the vocab was trained/exported without BERT special tokens.","commonSituations":"Reusing a raw WordPiece vocab (from ` trainers.WordPieceTrainer`) that never got special tokens added; adapting tokenizers across models (using XLNet-style tokens in a BERT wrapper); typos or casing differences between the `cls_token` argument and vocab entries.","solutions":["Pass the CLS token exactly as stored in the vocab: `BertWordPieceTokenizer(vocab_file, cls_token='<actual token>')`.","Verify membership before constructing: `assert cls_token in json.load(open(vocab_file))`.","Add `[CLS]` (and `[SEP]`) to the vocab file, or build the tokenizer manually with `Tokenizer(WordPiece(...))` and skip `BertProcessing` if you don't need BERT-style post-processing."],"exampleFix":"// before\ntok = BertWordPieceTokenizer(\"vocab.txt\", cls_token=\"[CLS]\")  # vocab lacks [CLS]\n// after\ntok = BertWordPieceTokenizer(\"vocab.txt\", cls_token=\"[CLS]\" if \"[CLS]\" in vocab else \"<cls>\")","handlingStrategy":"validation","validationCode":"import json\nvocab = json.load(open(vocab_file, encoding=\"utf-8\"))\nif cls_token not in vocab:\n    raise ValueError(f\"cls_token {cls_token!r} missing from vocab\")\ntok = BertWordPieceTokenizer(vocab_file, cls_token=cls_token)","typeGuard":"def vocab_has(vocab_file: str, token: str) -> bool:\n    import json\n    return token in json.load(open(vocab_file, encoding=\"utf-8\"))","tryCatchPattern":"try:\n    tok = BertWordPieceTokenizer(vocab_file, cls_token=cls_token)\nexcept TypeError as e:\n    if \"cls_token not found\" in str(e):\n        tok = BertWordPieceTokenizer(vocab_file, cls_token=find_cls(vocab_file))\n    else:\n        raise","preventionTips":["Check both CLS and SEP membership in the vocab together before construction.","Match casing exactly — [CLS] and [cls] are different vocab entries.","For non-BERT vocab files, build Tokenizer(WordPiece(...)) manually instead of using the BERT wrapper."],"tags":["python","tokenizer","vocabulary","config-mismatch"],"backgroundTag":"resource-not-found","analyzedSha":"6cfd9d385ca0ed91c10b49f0ce97d02cfde1b607","analyzedAt":"2026-09-09T11:43:25.027Z","contentChangedAt":"2026-09-09T11:43:25.027Z","schemaVersion":2},"datasetVersion":"2026-09-16T04:17:20.429Z"}