openai/whisper · error · KeyError

Language {language} not found in tokenizer.

Error message

Language {language} not found in tokenizer.

What it means

Tokenizer.to_language_token(language) looks up the special token <|language|> in the tokenizer's vocabulary. A KeyError means the language string itself is set, but no matching special token exists — typically because the language code is invalid, or the tokenizer was built with a reduced num_languages that excludes it (special token IDs beyond the multilingual vocabulary are absent).

Source

Thrown at whisper/tokenizer.py:223

        return self.special_tokens["<|notimestamps|>"]

    @cached_property
    def timestamp_begin(self) -> int:
        return self.special_tokens["<|0.00|>"]

    @cached_property
    def language_token(self) -> int:
        """Returns the token id corresponding to the value of the `language` field"""
        if self.language is None:
            raise ValueError("This tokenizer does not have language token configured")

        return self.to_language_token(self.language)

    def to_language_token(self, language):
        if token := self.special_tokens.get(f"<|{language}|>", None):
            return token

        raise KeyError(f"Language {language} not found in tokenizer.")

    @cached_property
    def all_language_tokens(self) -> Tuple[int]:
        result = []
        for token, token_id in self.special_tokens.items():
            if token.strip("<|>") in LANGUAGES:
                result.append(token_id)
        return tuple(result)[: self.num_languages]

    @cached_property
    def all_language_codes(self) -> Tuple[str]:
        return tuple(self.decode([_l]).strip("<|>") for _l in self.all_language_tokens)

    @cached_property
    def sot_sequence_including_notimestamps(self) -> Tuple[int]:
        return tuple(list(self.sot_sequence) + [self.no_timestamps])

    @cached_property

View on GitHub (pinned to 5f86d1d863)

Solutions

1. Use the canonical two-letter code from whisper.tokenizer.LANGUAGES values (e.g. 'en', 'de')
2. Rebuild the tokenizer with num_languages=model.dims.n_vocab matching the checkpoint (whisper.load_model does this automatically — prefer it)
3. Check membership first: f'<|{lang}|>' in tokenizer.special_tokens before calling

Example fix

# before
tok = whisper.tokenizer.get_tokenizer(True, language="english")
_ = tok.to_language_token("english")  # KeyError

# after
tok = whisper.tokenizer.get_tokenizer(True, language="en")
_ = tok.to_language_token("en")

def language_token_exists(tokenizer, language: str) -> bool:
    return f"<|{language}|>" in tokenizer.special_tokens

try:
    tid = tokenizer.to_language_token(lang)
except KeyError:
    raise ValueError(f"language {lang!r} unavailable in this checkpoint's vocabulary") from None

When it happens

Trigger: get_tokenizer(True, num_languages=50) then requesting a language whose token sits beyond the truncated vocabulary; calling to_language_token('xx') with an unknown code; language strings with whitespace/case issues (' EN ') that miss the '<|en|>' key.

Common situations: Fine-tuned custom checkpoints with fewer languages where model.dims.n_vocab was lowered; passing full language names ('english') where a code ('en') is required; stale tokenizer state after switching models in a long-running process.

Related errors

• Unsupported language: {language}
• {download_target} exists and is not a regular file
• Model {name} not found; available models = {available_models
• This model doesn't have language tokens so it can't perform
• This tokenizer does not have language token configured

AI-assisted analysis of openai/whisper@5f86d1d863 (2026-08-14). Data as JSON: /api/errors/a181a153e8b3abf4. Report an issue: GitHub.