What causes "cls_token not found in the vocabulary" in huggingface/tokenizers?

Pass the CLS token exactly as stored in the vocab: `BertWordPieceTokenizer(vocab_file, cls_token=' ')`.

huggingface/tokenizers · error · TypeError

cls_token not found in the vocabulary

Error message

cls_token not found in the vocabulary

What it means

`BertWordPieceTokenizer.__init__` raises this `TypeError` when a vocab file is provided but the configured `cls_token` (default `"[CLS]"`) is not present in the vocabulary. The BERT post-processor needs both SEP and CLS token ids, so the constructor validates CLS right after SEP and refuses to build an inconsistent tokenizer.

Solutions

Pass the CLS token exactly as stored in the vocab: `BertWordPieceTokenizer(vocab_file, cls_token='<actual token>')`.
Verify membership before constructing: `assert cls_token in json.load(open(vocab_file))`.
Add `[CLS]` (and `[SEP]`) to the vocab file, or build the tokenizer manually with `Tokenizer(WordPiece(...))` and skip `BertProcessing` if you don't need BERT-style post-processing.

Example fix

// before
tok = BertWordPieceTokenizer("vocab.txt", cls_token="[CLS]")  # vocab lacks [CLS]
// after
tok = BertWordPieceTokenizer("vocab.txt", cls_token="[CLS]" if "[CLS]" in vocab else "<cls>")

Defensive patterns

Strategy: validation

Validate before calling

import json
vocab = json.load(open(vocab_file, encoding="utf-8"))
if cls_token not in vocab:
    raise ValueError(f"cls_token {cls_token!r} missing from vocab")
tok = BertWordPieceTokenizer(vocab_file, cls_token=cls_token)

Type guard

def vocab_has(vocab_file: str, token: str) -> bool:
    import json
    return token in json.load(open(vocab_file, encoding="utf-8"))

Try / catch

try:
    tok = BertWordPieceTokenizer(vocab_file, cls_token=cls_token)
except TypeError as e:
    if "cls_token not found" in str(e):
        tok = BertWordPieceTokenizer(vocab_file, cls_token=find_cls(vocab_file))
    else:
        raise

Prevention

Check both CLS and SEP membership in the vocab together before construction.
Match casing exactly — [CLS] and [cls] are different vocab entries.
For non-BERT vocab files, build Tokenizer(WordPiece(...)) manually instead of using the BERT wrapper.

When it happens

Trigger: Constructing `BertWordPieceTokenizer(vocab_file, cls_token="[CLS]")` where the vocab lacks that exact string — e.g. vocab uses lowercase `[cls]`, another marker like `<s>`, or the vocab was trained/exported without BERT special tokens.

Common situations: Reusing a raw WordPiece vocab (from ` trainers.WordPieceTrainer`) that never got special tokens added; adapting tokenizers across models (using XLNet-style tokens in a BERT wrapper); typos or casing differences between the `cls_token` argument and vocab entries.

Understand the failure class

Background: 'Could not be found', 'does not exist', 'not found in database': the resource-not-found family when an ID, slug, key, or URI lookup comes back empty — this error's family across 20 libraries.

Related errors

AI-assisted analysis of huggingface/tokenizers@6cfd9d385c (2026-09-09). Data as JSON: /api/errors/c82d4ce2ac4b2d74. Report an issue: GitHub.

Appendix: source

Thrown at bindings/python/py_src/tokenizers/implementations/bert_wordpiece.py:60

            tokenizer.add_special_tokens([str(pad_token)])
        if tokenizer.token_to_id(str(mask_token)) is not None:
            tokenizer.add_special_tokens([str(mask_token)])

        tokenizer.normalizer = BertNormalizer(
            clean_text=clean_text,
            handle_chinese_chars=handle_chinese_chars,
            strip_accents=strip_accents,
            lowercase=lowercase,
        )
        tokenizer.pre_tokenizer = BertPreTokenizer()

        if vocab is not None:
            sep_token_id = tokenizer.token_to_id(str(sep_token))
            if sep_token_id is None:
                raise TypeError("sep_token not found in the vocabulary")
            cls_token_id = tokenizer.token_to_id(str(cls_token))
            if cls_token_id is None:
                raise TypeError("cls_token not found in the vocabulary")

            tokenizer.post_processor = BertProcessing((str(sep_token), sep_token_id), (str(cls_token), cls_token_id))
        tokenizer.decoder = decoders.WordPiece(prefix=wordpieces_prefix)

        parameters = {
            "model": "BertWordPiece",
            "unk_token": unk_token,
            "sep_token": sep_token,
            "cls_token": cls_token,
            "pad_token": pad_token,
            "mask_token": mask_token,
            "clean_text": clean_text,
            "handle_chinese_chars": handle_chinese_chars,
            "strip_accents": strip_accents,
            "lowercase": lowercase,
            "wordpieces_prefix": wordpieces_prefix,
        }

View on GitHub (pinned to 6cfd9d385c)