{"record":{"id":"d386e272035d0a05","repo":"openai/whisper","slug":"this-tokenizer-does-not-have-language-token-config","errorCode":null,"errorMessage":"This tokenizer does not have language token configured","messagePattern":"This tokenizer does not have language token configured","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"whisper/tokenizer.py","lineNumber":215,"sourceCode":"        return self.special_tokens[\"<|startofprev|>\"]\n\n    @cached_property\n    def no_speech(self) -> int:\n        return self.special_tokens[\"<|nospeech|>\"]\n\n    @cached_property\n    def no_timestamps(self) -> int:\n        return self.special_tokens[\"<|notimestamps|>\"]\n\n    @cached_property\n    def timestamp_begin(self) -> int:\n        return self.special_tokens[\"<|0.00|>\"]\n\n    @cached_property\n    def language_token(self) -> int:\n        \"\"\"Returns the token id corresponding to the value of the `language` field\"\"\"\n        if self.language is None:\n            raise ValueError(\"This tokenizer does not have language token configured\")\n\n        return self.to_language_token(self.language)\n\n    def to_language_token(self, language):\n        if token := self.special_tokens.get(f\"<|{language}|>\", None):\n            return token\n\n        raise KeyError(f\"Language {language} not found in tokenizer.\")\n\n    @cached_property\n    def all_language_tokens(self) -> Tuple[int]:\n        result = []\n        for token, token_id in self.special_tokens.items():\n            if token.strip(\"<|>\") in LANGUAGES:\n                result.append(token_id)\n        return tuple(result)[: self.num_languages]\n\n    @cached_property","sourceCodeStart":197,"sourceCodeEnd":233,"githubUrl":"https://github.com/openai/whisper/blob/5f86d1d86363843179951550570367b37c5d6f78/whisper/tokenizer.py#L197-L233","documentation":"Tokenizer.language_token is a cached_property that requires the tokenizer to have been constructed with a language. Tokenizers built for English-only models (get_tokenizer(multilingual=False), or automatically for *.en checkpoints) set language=None, so accessing language_token (directly, or via detect_language / DecodingTask prompts) raises ValueError.","triggerScenarios":"tokenizer.language_token on a tokenizer created with multilingual=False; whisper.detect_language() on an .en model (same root cause as the lang-id error, surfacing here in the tokenizer layer); building DecodingOptions with language set but an English tokenizer.","commonSituations":"Writing generic code that queries language_token for every model without checking is_multilingual; loading '.en' checkpoints in pipelines that also serve multilingual models.","solutions":["Use a multilingual model/tokenizer: whisper.load_model('base') rather than 'base.en'","Guard with `if tokenizer.is_multilingual:` (or model.is_multilingual) before touching language_token","Pass language explicitly to transcribe() instead of reading it from the tokenizer"],"exampleFix":"# before\n_ = tokenizer.language_token  # ValueError when tokenizer.language is None\n\n# after\nif tokenizer.is_multilingual:\n    _ = tokenizer.language_token\nelse:\n    _ = tokenizer.sot  # English-only: no language token exists","handlingStrategy":"type-guard","validationCode":"def tokenizer_has_language_token(tokenizer) -> bool:\n    return tokenizer.is_multilingual and tokenizer.language is not None","typeGuard":"def is_multilingual_tokenizer(tok) -> bool:\n    return bool(getattr(tok, \"is_multilingual\", False)) and tok.language is not None","tryCatchPattern":"try:\n    lang_id = tokenizer.language_token\nexcept ValueError:\n    lang_id = None  # English-only tokenizer; skip language-prompt logic","preventionTips":["Check tokenizer.is_multilingual before reading language_token or building sot prompts","Construct tokenizers via get_tokenizer(model.is_multilingual, num_languages=model.num_languages) so state matches the checkpoint"],"tags":["tokenizer","language-id","multilingual","model-config"],"backgroundTag":null,"analyzedSha":"5f86d1d86363843179951550570367b37c5d6f78","analyzedAt":"2026-08-14T18:53:59.547Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}