huggingface/tokenizers · error · TypeError

sep_token not found in the vocabulary

Error message

sep_token not found in the vocabulary

What it means

`BertWordPieceTokenizer.__init__` raises this `TypeError` when a vocab file is provided but the configured `sep_token` (default `"[SEP]"`) cannot be found via `token_to_id` in that vocabulary. BERT's post-processor requires the SEP token and its id to build `BertProcessing`, so a missing SEP makes the tokenizer configuration invalid.

Solutions

1. Pass the SEP token exactly as it appears in the vocab: `BertWordPieceTokenizer(vocab_file, sep_token='<actual token in vocab>')`.
2. Inspect the vocab to confirm the token: `"[SEP]" in json.load(open(vocab_file))`.
3. Add the missing special tokens to the vocab file, or set `sep_token=None`-compatible handling by constructing a plain `Tokenizer` with `WordPiece` manually if no BERT post-processing is wanted.

Example fix

// before
tok = BertWordPieceTokenizer("vocab.txt", sep_token="[SEP]")  # vocab has no [SEP]
// after
tok = BertWordPieceTokenizer("vocab.txt", sep_token="</s>")  # token present in this vocab

import json
vocab = json.load(open(vocab_file, encoding="utf-8"))
if sep_token not in vocab:
    raise ValueError(f"sep_token {sep_token!r} missing from vocab")
tok = BertWordPieceTokenizer(vocab_file, sep_token=sep_token)

def vocab_has(vocab_file: str, token: str) -> bool:
    import json
    return token in json.load(open(vocab_file, encoding="utf-8"))

try:
    tok = BertWordPieceTokenizer(vocab_file, sep_token=sep_token)
except TypeError as e:
    if "sep_token not found" in str(e):
        # fall back to a token that exists in the vocab
        tok = BertWordPieceTokenizer(vocab_file, sep_token=find_sep(vocab_file))
    else:
        raise

When it happens

Trigger: Constructing `BertWordPieceTokenizer(vocab_file, sep_token="[SEP]")` with a vocab that lacks the exact string (case/whitespace mismatch, custom `[sep]`, German BERT using different tokens, or building a vocab programmatically and forgetting to add `[SEP]`).

Common situations: Using a WordPiece vocab scraped from a model repo that stores tokens without brackets; renaming special tokens to lowercase (`sep_token="[sep]"`) while the vocab has `[SEP]`; training a WordPiece model and exporting vocab without including BERT special tokens.

Understand the failure class

Background: 'Could not be found', 'does not exist', 'not found in database': the resource-not-found family when an ID, slug, key, or URI lookup comes back empty — this error's family across 20 libraries.

Related errors

• cls_token not found in the vocabulary
• encode: `sequence` can't be `None`
• encode_batch: `inputs` can't be `None`
• async_encode_batch: `inputs` can't be `None`
• async_encode_batch_fast: `inputs` can't be `None`

AI-assisted analysis of huggingface/tokenizers@6cfd9d385c (2026-09-09). Data as JSON: /api/errors/79f7cc726cb02a3c. Report an issue: GitHub.

Appendix: source

Thrown at bindings/python/py_src/tokenizers/implementations/bert_wordpiece.py:57

        if tokenizer.token_to_id(str(cls_token)) is not None:
            tokenizer.add_special_tokens([str(cls_token)])
        if tokenizer.token_to_id(str(pad_token)) is not None:
            tokenizer.add_special_tokens([str(pad_token)])
        if tokenizer.token_to_id(str(mask_token)) is not None:
            tokenizer.add_special_tokens([str(mask_token)])

        tokenizer.normalizer = BertNormalizer(
            clean_text=clean_text,
            handle_chinese_chars=handle_chinese_chars,
            strip_accents=strip_accents,
            lowercase=lowercase,
        )
        tokenizer.pre_tokenizer = BertPreTokenizer()

        if vocab is not None:
            sep_token_id = tokenizer.token_to_id(str(sep_token))
            if sep_token_id is None:
                raise TypeError("sep_token not found in the vocabulary")
            cls_token_id = tokenizer.token_to_id(str(cls_token))
            if cls_token_id is None:
                raise TypeError("cls_token not found in the vocabulary")

            tokenizer.post_processor = BertProcessing((str(sep_token), sep_token_id), (str(cls_token), cls_token_id))
        tokenizer.decoder = decoders.WordPiece(prefix=wordpieces_prefix)

        parameters = {
            "model": "BertWordPiece",
            "unk_token": unk_token,
            "sep_token": sep_token,
            "cls_token": cls_token,
            "pad_token": pad_token,
            "mask_token": mask_token,
            "clean_text": clean_text,
            "handle_chinese_chars": handle_chinese_chars,
            "strip_accents": strip_accents,
            "lowercase": lowercase,

View on GitHub (pinned to 6cfd9d385c)