{"record":{"id":"a181a153e8b3abf4","repo":"openai/whisper","slug":"language-language-not-found-in-tokenizer","errorCode":null,"errorMessage":"Language {language} not found in tokenizer.","messagePattern":"Language (.+?) not found in tokenizer\\.","errorType":"exception","errorClass":"KeyError","httpStatus":null,"severity":"error","filePath":"whisper/tokenizer.py","lineNumber":223,"sourceCode":"        return self.special_tokens[\"<|notimestamps|>\"]\n\n    @cached_property\n    def timestamp_begin(self) -> int:\n        return self.special_tokens[\"<|0.00|>\"]\n\n    @cached_property\n    def language_token(self) -> int:\n        \"\"\"Returns the token id corresponding to the value of the `language` field\"\"\"\n        if self.language is None:\n            raise ValueError(\"This tokenizer does not have language token configured\")\n\n        return self.to_language_token(self.language)\n\n    def to_language_token(self, language):\n        if token := self.special_tokens.get(f\"<|{language}|>\", None):\n            return token\n\n        raise KeyError(f\"Language {language} not found in tokenizer.\")\n\n    @cached_property\n    def all_language_tokens(self) -> Tuple[int]:\n        result = []\n        for token, token_id in self.special_tokens.items():\n            if token.strip(\"<|>\") in LANGUAGES:\n                result.append(token_id)\n        return tuple(result)[: self.num_languages]\n\n    @cached_property\n    def all_language_codes(self) -> Tuple[str]:\n        return tuple(self.decode([_l]).strip(\"<|>\") for _l in self.all_language_tokens)\n\n    @cached_property\n    def sot_sequence_including_notimestamps(self) -> Tuple[int]:\n        return tuple(list(self.sot_sequence) + [self.no_timestamps])\n\n    @cached_property","sourceCodeStart":205,"sourceCodeEnd":241,"githubUrl":"https://github.com/openai/whisper/blob/5f86d1d86363843179951550570367b37c5d6f78/whisper/tokenizer.py#L205-L241","documentation":"Tokenizer.to_language_token(language) looks up the special token <|language|> in the tokenizer's vocabulary. A KeyError means the language string itself is set, but no matching special token exists — typically because the language code is invalid, or the tokenizer was built with a reduced num_languages that excludes it (special token IDs beyond the multilingual vocabulary are absent).","triggerScenarios":"get_tokenizer(True, num_languages=50) then requesting a language whose token sits beyond the truncated vocabulary; calling to_language_token('xx') with an unknown code; language strings with whitespace/case issues (' EN ') that miss the '<|en|>' key.","commonSituations":"Fine-tuned custom checkpoints with fewer languages where model.dims.n_vocab was lowered; passing full language names ('english') where a code ('en') is required; stale tokenizer state after switching models in a long-running process.","solutions":["Use the canonical two-letter code from whisper.tokenizer.LANGUAGES values (e.g. 'en', 'de')","Rebuild the tokenizer with num_languages=model.dims.n_vocab matching the checkpoint (whisper.load_model does this automatically — prefer it)","Check membership first: f'<|{lang}|>' in tokenizer.special_tokens before calling"],"exampleFix":"# before\ntok = whisper.tokenizer.get_tokenizer(True, language=\"english\")\n_ = tok.to_language_token(\"english\")  # KeyError\n\n# after\ntok = whisper.tokenizer.get_tokenizer(True, language=\"en\")\n_ = tok.to_language_token(\"en\")","handlingStrategy":"validation","validationCode":"def language_token_exists(tokenizer, language: str) -> bool:\n    return f\"<|{language}|>\" in tokenizer.special_tokens","typeGuard":null,"tryCatchPattern":"try:\n    tid = tokenizer.to_language_token(lang)\nexcept KeyError:\n    raise ValueError(f\"language {lang!r} unavailable in this checkpoint's vocabulary\") from None","preventionTips":["Restrict language inputs to codes present in whisper.tokenizer.LANGUAGES","After building a tokenizer for fine-tuned checkpoints, assert your target languages exist in special_tokens"],"tags":["tokenizer","language","vocabulary","configuration"],"backgroundTag":null,"analyzedSha":"5f86d1d86363843179951550570367b37c5d6f78","analyzedAt":"2026-08-14T18:53:59.547Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}