openai/whisper · error · ValueError

This tokenizer does not have language token configured

Error message

This tokenizer does not have language token configured

What it means

Tokenizer.language_token is a cached_property that requires the tokenizer to have been constructed with a language. Tokenizers built for English-only models (get_tokenizer(multilingual=False), or automatically for *.en checkpoints) set language=None, so accessing language_token (directly, or via detect_language / DecodingTask prompts) raises ValueError.

Source

Thrown at whisper/tokenizer.py:215

        return self.special_tokens["<|startofprev|>"]

    @cached_property
    def no_speech(self) -> int:
        return self.special_tokens["<|nospeech|>"]

    @cached_property
    def no_timestamps(self) -> int:
        return self.special_tokens["<|notimestamps|>"]

    @cached_property
    def timestamp_begin(self) -> int:
        return self.special_tokens["<|0.00|>"]

    @cached_property
    def language_token(self) -> int:
        """Returns the token id corresponding to the value of the `language` field"""
        if self.language is None:
            raise ValueError("This tokenizer does not have language token configured")

        return self.to_language_token(self.language)

    def to_language_token(self, language):
        if token := self.special_tokens.get(f"<|{language}|>", None):
            return token

        raise KeyError(f"Language {language} not found in tokenizer.")

    @cached_property
    def all_language_tokens(self) -> Tuple[int]:
        result = []
        for token, token_id in self.special_tokens.items():
            if token.strip("<|>") in LANGUAGES:
                result.append(token_id)
        return tuple(result)[: self.num_languages]

    @cached_property

View on GitHub (pinned to 5f86d1d863)

Solutions

1. Use a multilingual model/tokenizer: whisper.load_model('base') rather than 'base.en'
2. Guard with `if tokenizer.is_multilingual:` (or model.is_multilingual) before touching language_token
3. Pass language explicitly to transcribe() instead of reading it from the tokenizer

Example fix

# before
_ = tokenizer.language_token  # ValueError when tokenizer.language is None

# after
if tokenizer.is_multilingual:
    _ = tokenizer.language_token
else:
    _ = tokenizer.sot  # English-only: no language token exists

def tokenizer_has_language_token(tokenizer) -> bool:
    return tokenizer.is_multilingual and tokenizer.language is not None

def is_multilingual_tokenizer(tok) -> bool:
    return bool(getattr(tok, "is_multilingual", False)) and tok.language is not None

try:
    lang_id = tokenizer.language_token
except ValueError:
    lang_id = None  # English-only tokenizer; skip language-prompt logic

When it happens

Trigger: tokenizer.language_token on a tokenizer created with multilingual=False; whisper.detect_language() on an .en model (same root cause as the lang-id error, surfacing here in the tokenizer layer); building DecodingOptions with language set but an English tokenizer.

Common situations: Writing generic code that queries language_token for every model without checking is_multilingual; loading '.en' checkpoints in pipelines that also serve multilingual models.

Related errors

• This model doesn't have language tokens so it can't perform
• Language {language} not found in tokenizer.
• Unsupported language: {language}

AI-assisted analysis of openai/whisper@5f86d1d863 (2026-08-14). Data as JSON: /api/errors/d386e272035d0a05. Report an issue: GitHub.