{"record":{"id":"79f7cc726cb02a3c","repo":"huggingface/tokenizers","slug":"sep-token-not-found-in-the-vocabulary","errorCode":null,"errorMessage":"sep_token not found in the vocabulary","messagePattern":"sep_token not found in the vocabulary","errorType":"exception","errorClass":"TypeError","httpStatus":null,"severity":"error","filePath":"bindings/python/py_src/tokenizers/implementations/bert_wordpiece.py","lineNumber":57,"sourceCode":"        if tokenizer.token_to_id(str(cls_token)) is not None:\n            tokenizer.add_special_tokens([str(cls_token)])\n        if tokenizer.token_to_id(str(pad_token)) is not None:\n            tokenizer.add_special_tokens([str(pad_token)])\n        if tokenizer.token_to_id(str(mask_token)) is not None:\n            tokenizer.add_special_tokens([str(mask_token)])\n\n        tokenizer.normalizer = BertNormalizer(\n            clean_text=clean_text,\n            handle_chinese_chars=handle_chinese_chars,\n            strip_accents=strip_accents,\n            lowercase=lowercase,\n        )\n        tokenizer.pre_tokenizer = BertPreTokenizer()\n\n        if vocab is not None:\n            sep_token_id = tokenizer.token_to_id(str(sep_token))\n            if sep_token_id is None:\n                raise TypeError(\"sep_token not found in the vocabulary\")\n            cls_token_id = tokenizer.token_to_id(str(cls_token))\n            if cls_token_id is None:\n                raise TypeError(\"cls_token not found in the vocabulary\")\n\n            tokenizer.post_processor = BertProcessing((str(sep_token), sep_token_id), (str(cls_token), cls_token_id))\n        tokenizer.decoder = decoders.WordPiece(prefix=wordpieces_prefix)\n\n        parameters = {\n            \"model\": \"BertWordPiece\",\n            \"unk_token\": unk_token,\n            \"sep_token\": sep_token,\n            \"cls_token\": cls_token,\n            \"pad_token\": pad_token,\n            \"mask_token\": mask_token,\n            \"clean_text\": clean_text,\n            \"handle_chinese_chars\": handle_chinese_chars,\n            \"strip_accents\": strip_accents,\n            \"lowercase\": lowercase,","sourceCodeStart":39,"sourceCodeEnd":75,"githubUrl":"https://github.com/huggingface/tokenizers/blob/6cfd9d385ca0ed91c10b49f0ce97d02cfde1b607/bindings/python/py_src/tokenizers/implementations/bert_wordpiece.py#L39-L75","documentation":"`BertWordPieceTokenizer.__init__` raises this `TypeError` when a vocab file is provided but the configured `sep_token` (default `\"[SEP]\"`) cannot be found via `token_to_id` in that vocabulary. BERT's post-processor requires the SEP token and its id to build `BertProcessing`, so a missing SEP makes the tokenizer configuration invalid.","triggerScenarios":"Constructing `BertWordPieceTokenizer(vocab_file, sep_token=\"[SEP]\")` with a vocab that lacks the exact string (case/whitespace mismatch, custom `[sep]`, German BERT using different tokens, or building a vocab programmatically and forgetting to add `[SEP]`).","commonSituations":"Using a WordPiece vocab scraped from a model repo that stores tokens without brackets; renaming special tokens to lowercase (`sep_token=\"[sep]\"`) while the vocab has `[SEP]`; training a WordPiece model and exporting vocab without including BERT special tokens.","solutions":["Pass the SEP token exactly as it appears in the vocab: `BertWordPieceTokenizer(vocab_file, sep_token='<actual token in vocab>')`.","Inspect the vocab to confirm the token: `\"[SEP]\" in json.load(open(vocab_file))`.","Add the missing special tokens to the vocab file, or set `sep_token=None`-compatible handling by constructing a plain `Tokenizer` with `WordPiece` manually if no BERT post-processing is wanted."],"exampleFix":"// before\ntok = BertWordPieceTokenizer(\"vocab.txt\", sep_token=\"[SEP]\")  # vocab has no [SEP]\n// after\ntok = BertWordPieceTokenizer(\"vocab.txt\", sep_token=\"</s>\")  # token present in this vocab","handlingStrategy":"validation","validationCode":"import json\nvocab = json.load(open(vocab_file, encoding=\"utf-8\"))\nif sep_token not in vocab:\n    raise ValueError(f\"sep_token {sep_token!r} missing from vocab\")\ntok = BertWordPieceTokenizer(vocab_file, sep_token=sep_token)","typeGuard":"def vocab_has(vocab_file: str, token: str) -> bool:\n    import json\n    return token in json.load(open(vocab_file, encoding=\"utf-8\"))","tryCatchPattern":"try:\n    tok = BertWordPieceTokenizer(vocab_file, sep_token=sep_token)\nexcept TypeError as e:\n    if \"sep_token not found\" in str(e):\n        # fall back to a token that exists in the vocab\n        tok = BertWordPieceTokenizer(vocab_file, sep_token=find_sep(vocab_file))\n    else:\n        raise","preventionTips":["Always verify special tokens exist in the vocab before constructing BERT tokenizers.","Copy token strings directly from the vocab file instead of retyping them.","When training a WordPiece model, add [CLS]/[SEP] to the vocab before wrapping with BertWordPieceTokenizer."],"tags":["python","tokenizer","vocabulary","config-mismatch"],"backgroundTag":"resource-not-found","analyzedSha":"6cfd9d385ca0ed91c10b49f0ce97d02cfde1b607","analyzedAt":"2026-09-09T11:43:25.027Z","contentChangedAt":"2026-09-09T11:43:25.027Z","schemaVersion":2},"datasetVersion":"2026-09-16T04:17:20.429Z"}