{"record":{"id":"df182870665947ba","repo":"immich-app/immich","slug":"pad-token-pad-token-not-found-in-tokenizer-voc","errorCode":null,"errorMessage":"Pad token '{pad_token}' not found in tokenizer vocab","messagePattern":"Pad token '(.+?)' not found in tokenizer vocab","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"machine-learning/immich_ml/models/clip/textual.py","lineNumber":94,"sourceCode":"\n    @cached_property\n    def tokenizer_cfg(self) -> dict[str, Any]:\n        log.debug(f\"Loading tokenizer config for CLIP model '{self.model_name}'\")\n        tokenizer_cfg: dict[str, Any] = json.load(self.tokenizer_cfg_path.open())\n        log.debug(f\"Loaded tokenizer config for CLIP model '{self.model_name}'\")\n        return tokenizer_cfg\n\n\nclass OpenClipTextualEncoder(BaseCLIPTextualEncoder):\n    def _load_tokenizer(self) -> Tokenizer:\n        context_length: int = self.text_cfg.get(\"context_length\", 77)\n        pad_token: str = self.tokenizer_cfg[\"pad_token\"]\n\n        tokenizer: Tokenizer = Tokenizer.from_file(self.tokenizer_file_path.as_posix())\n\n        pad_id = tokenizer.token_to_id(pad_token)\n        if pad_id is None:\n            raise ValueError(f\"Pad token '{pad_token}' not found in tokenizer vocab\")\n        tokenizer.enable_padding(length=context_length, pad_token=pad_token, pad_id=pad_id)\n        tokenizer.enable_truncation(max_length=context_length)\n\n        return tokenizer\n\n    def tokenize(self, text: str, language: str | None = None) -> dict[str, NDArray[np.int32]]:\n        text = clean_text(text, canonicalize=self.canonicalize)\n        if self.is_nllb and language is not None:\n            flores_code = WEBLATE_TO_FLORES200.get(language)\n            if flores_code is None:\n                no_country = language.split(\"-\")[0]\n                flores_code = WEBLATE_TO_FLORES200.get(no_country)\n                if flores_code is None:\n                    log.warning(f\"Language '{language}' not found, defaulting to 'en'\")\n                    flores_code = \"eng_Latn\"\n            text = f\"{flores_code}{text}\"\n        tokens: Encoding = self.tokenizer.encode(text)\n        return {\"text\": np.array([tokens.ids], dtype=np.int32)}","sourceCodeStart":76,"sourceCodeEnd":112,"githubUrl":"https://github.com/immich-app/immich/blob/199723261c6ffa897fec8ccdaea6359e39c37cc3/machine-learning/immich_ml/models/clip/textual.py#L76-L112","documentation":"Raised by OpenClipTextualEncoder._load_tokenizer when the pad_token string declared in tokenizer_config.json has no corresponding entry in the vocab of tokenizer.json (Tokenizer.token_to_id returns None). The pad token must exist so enable_padding can emit valid pad_id values for every position it inserts.","triggerScenarios":"Loading a CLIP/OpenCLIP textual encoder whose tokenizer_config.json and tokenizer.json come from different model revisions or were swapped; a custom tokenizer.json whose vocab was pruned; a pad_token value with whitespace/casing differences versus the vocab (e.g. '<pad>' vs '<pad >'); pointing cache_dir at a hand-edited tokenizer file.","commonSituations":"Partial or mixed snapshot_download leaving an old tokenizer_config.json with a new tokenizer.json; using a community/fine-tuned model whose tokenizer files are inconsistent; encoding artifacts (BOM, escaped quotes) in tokenizer_config.json changing the pad_token string; model repo renamed the pad token between releases.","solutions":["Open tokenizer_config.json and note the exact pad_token string (including angle brackets and spaces).","Open tokenizer.json's vocab and confirm that exact string is a key; check for case or whitespace mismatch.","Call clear_cache() and re-download to get a consistent pair of tokenizer files from the same revision.","If the repo is genuinely inconsistent, pin a known-good revision of the model or override tokenizer_cfg with a corrected pad_token present in the vocab.","Add a startup self-check in your deployment that asserts token_to_id(pad_token) is not None before serving requests."],"exampleFix":"# before\npad_token: str = self.tokenizer_cfg['pad_token']           # e.g. '<pad >' (typo in config)\npad_id = tokenizer.token_to_id(pad_token)                  # None -> ValueError\n\n# after\npad_token: str = self.tokenizer_cfg['pad_token']\npad_id = tokenizer.token_to_id(pad_token)\nif pad_id is None:\n    # fall back to a vocab-known pad token before failing\n    for candidate in ('<|endoftext|>', '</s>', '<pad>'):\n        pad_id = tokenizer.token_to_id(candidate)\n        if pad_id is not None:\n            pad_token = candidate\n            break\n    if pad_id is None:\n        raise ValueError(f\"Pad token '{pad_token}' not found in tokenizer vocab\")","handlingStrategy":"validation","validationCode":"from tokenizers import Tokenizer\n\ndef validate_pad_token(tokenizer_file_path: str, pad_token: str) -> int:\n    tok = Tokenizer.from_file(tokenizer_file_path)\n    pad_id = tok.token_to_id(pad_token)\n    if pad_id is None:\n        raise ValueError(\n            f\"Pad token {pad_token!r} not in vocab of {tokenizer_file_path}; \"\n            f\"check tokenizer_config.json vs tokenizer.json\"\n        )\n    return pad_id\n\n# call before instantiating the encoder:\nvalidate_pad_token(str(tokenizer_file_path), tokenizer_cfg['pad_token'])","typeGuard":"def tokenizer_pad_token_is_known(tokenizer_file_path: str, pad_token: str) -> bool:\n    from tokenizers import Tokenizer\n    return Tokenizer.from_file(tokenizer_file_path).token_to_id(pad_token) is not None","tryCatchPattern":"try:\n    self.tokenizer = self._load_tokenizer()\nexcept ValueError as e:\n    if 'not found in tokenizer vocab' in str(e):\n        log.error(\"Tokenizer/config mismatch for %s; re-downloading model cache\", self.model_name)\n        self.clear_cache()\n        self.download()\n        self.tokenizer = self._load_tokenizer()\n    else:\n        raise","preventionTips":["Treat tokenizer_config.json and tokenizer.json as a pair: never replace one without the other.","Pin model revisions in snapshot_download so a later repo change cannot desync the files.","Add a startup health check that loads every CLIP encoder's tokenizer and asserts the pad token resolves."],"tags":["tokenizer","clip","config-mismatch","nlp"],"backgroundTag":null,"analyzedSha":"199723261c6ffa897fec8ccdaea6359e39c37cc3","analyzedAt":"2026-08-12T04:54:27.085Z","schemaVersion":2},"datasetVersion":"2026-08-12T13:17:24.610Z"}