immich-app/immich · error · ValueError

Pad token '{pad_token}' not found in tokenizer vocab

Error message

Pad token '{pad_token}' not found in tokenizer vocab

What it means

Raised by OpenClipTextualEncoder._load_tokenizer when the pad_token string declared in tokenizer_config.json has no corresponding entry in the vocab of tokenizer.json (Tokenizer.token_to_id returns None). The pad token must exist so enable_padding can emit valid pad_id values for every position it inserts.

Source

Thrown at machine-learning/immich_ml/models/clip/textual.py:94

    @cached_property
    def tokenizer_cfg(self) -> dict[str, Any]:
        log.debug(f"Loading tokenizer config for CLIP model '{self.model_name}'")
        tokenizer_cfg: dict[str, Any] = json.load(self.tokenizer_cfg_path.open())
        log.debug(f"Loaded tokenizer config for CLIP model '{self.model_name}'")
        return tokenizer_cfg


class OpenClipTextualEncoder(BaseCLIPTextualEncoder):
    def _load_tokenizer(self) -> Tokenizer:
        context_length: int = self.text_cfg.get("context_length", 77)
        pad_token: str = self.tokenizer_cfg["pad_token"]

        tokenizer: Tokenizer = Tokenizer.from_file(self.tokenizer_file_path.as_posix())

        pad_id = tokenizer.token_to_id(pad_token)
        if pad_id is None:
            raise ValueError(f"Pad token '{pad_token}' not found in tokenizer vocab")
        tokenizer.enable_padding(length=context_length, pad_token=pad_token, pad_id=pad_id)
        tokenizer.enable_truncation(max_length=context_length)

        return tokenizer

    def tokenize(self, text: str, language: str | None = None) -> dict[str, NDArray[np.int32]]:
        text = clean_text(text, canonicalize=self.canonicalize)
        if self.is_nllb and language is not None:
            flores_code = WEBLATE_TO_FLORES200.get(language)
            if flores_code is None:
                no_country = language.split("-")[0]
                flores_code = WEBLATE_TO_FLORES200.get(no_country)
                if flores_code is None:
                    log.warning(f"Language '{language}' not found, defaulting to 'en'")
                    flores_code = "eng_Latn"
            text = f"{flores_code}{text}"
        tokens: Encoding = self.tokenizer.encode(text)
        return {"text": np.array([tokens.ids], dtype=np.int32)}

View on GitHub (pinned to 199723261c)

Solutions

  1. Open tokenizer_config.json and note the exact pad_token string (including angle brackets and spaces).
  2. Open tokenizer.json's vocab and confirm that exact string is a key; check for case or whitespace mismatch.
  3. Call clear_cache() and re-download to get a consistent pair of tokenizer files from the same revision.
  4. If the repo is genuinely inconsistent, pin a known-good revision of the model or override tokenizer_cfg with a corrected pad_token present in the vocab.
  5. Add a startup self-check in your deployment that asserts token_to_id(pad_token) is not None before serving requests.

Example fix

# before
pad_token: str = self.tokenizer_cfg['pad_token']           # e.g. '<pad >' (typo in config)
pad_id = tokenizer.token_to_id(pad_token)                  # None -> ValueError

# after
pad_token: str = self.tokenizer_cfg['pad_token']
pad_id = tokenizer.token_to_id(pad_token)
if pad_id is None:
    # fall back to a vocab-known pad token before failing
    for candidate in ('<|endoftext|>', '</s>', '<pad>'):
        pad_id = tokenizer.token_to_id(candidate)
        if pad_id is not None:
            pad_token = candidate
            break
    if pad_id is None:
        raise ValueError(f"Pad token '{pad_token}' not found in tokenizer vocab")
Defensive patterns

Strategy: validation

Validate before calling

from tokenizers import Tokenizer

def validate_pad_token(tokenizer_file_path: str, pad_token: str) -> int:
    tok = Tokenizer.from_file(tokenizer_file_path)
    pad_id = tok.token_to_id(pad_token)
    if pad_id is None:
        raise ValueError(
            f"Pad token {pad_token!r} not in vocab of {tokenizer_file_path}; "
            f"check tokenizer_config.json vs tokenizer.json"
        )
    return pad_id

# call before instantiating the encoder:
validate_pad_token(str(tokenizer_file_path), tokenizer_cfg['pad_token'])

Type guard

def tokenizer_pad_token_is_known(tokenizer_file_path: str, pad_token: str) -> bool:
    from tokenizers import Tokenizer
    return Tokenizer.from_file(tokenizer_file_path).token_to_id(pad_token) is not None

Try / catch

try:
    self.tokenizer = self._load_tokenizer()
except ValueError as e:
    if 'not found in tokenizer vocab' in str(e):
        log.error("Tokenizer/config mismatch for %s; re-downloading model cache", self.model_name)
        self.clear_cache()
        self.download()
        self.tokenizer = self._load_tokenizer()
    else:
        raise

Prevention

When it happens

Trigger: Loading a CLIP/OpenCLIP textual encoder whose tokenizer_config.json and tokenizer.json come from different model revisions or were swapped; a custom tokenizer.json whose vocab was pruned; a pad_token value with whitespace/casing differences versus the vocab (e.g. '<pad>' vs '<pad >'); pointing cache_dir at a hand-edited tokenizer file.

Common situations: Partial or mixed snapshot_download leaving an old tokenizer_config.json with a new tokenizer.json; using a community/fine-tuned model whose tokenizer files are inconsistent; encoding artifacts (BOM, escaped quotes) in tokenizer_config.json changing the pad_token string; model repo renamed the pad token between releases.

Related errors

AI-assisted analysis of immich-app/immich@199723261c (2026-08-12). Data as JSON: /api/errors/df182870665947ba. Report an issue: GitHub.