openai/whisper · warning · ValueError

Unexpected token: {current}

Error message

Unexpected token: {current}

What it means

EnglishNumberNormalizer.postprocess_steps iterates over tokens produced by its own num2words-based tokenization of spoken numbers. The first 'Unexpected token' branch fires inside the numeric-word handling chain (current is a known numeric token class but falls through all sub-branches: not a units/teens/tens word covered, not 'point' with a decimal following). It signals a tokenizer-state edge case in transcribed ASR text rather than user misconfiguration.

Source

Thrown at whisper/normalizers/english.py:380

                        if value is not None:
                            yield output(value)
                        yield output(current)
                elif current == "double" or current == "triple":
                    if next in self.ones or next in self.zeros:
                        repeats = 2 if current == "double" else 3
                        ones = self.ones.get(next, 0)
                        value = str(value or "") + str(ones) * repeats
                        skip = True
                    else:
                        if value is not None:
                            yield output(value)
                        yield output(current)
                elif current == "point":
                    if next in self.decimals or next_is_numeric:
                        value = str(value or "") + "."
                else:
                    # should all have been covered at this point
                    raise ValueError(f"Unexpected token: {current}")
            else:
                # all should have been covered at this point
                raise ValueError(f"Unexpected token: {current}")

        if value is not None:
            yield output(value)

    def preprocess(self, s: str):
        # replace "<number> and a half" with "<number> point five"
        results = []

        segments = re.split(r"\band\s+a\s+half\b", s)
        for i, segment in enumerate(segments):
            if len(segment.strip()) == 0:
                continue
            if i == len(segments) - 1:
                results.append(segment)
            else:

View on GitHub (pinned to 5f86d1d863)

Solutions

1. Upgrade openai-whisper — the English normalizer has had multiple fixes for unexpected-token inputs
2. Wrap normalization in try/except and fall back to the raw/un-normalized text for that segment
3. Reproduce with EnglishTextNormalizer(s) directly to isolate the offending substring, then pre-clean it (e.g. strip stray number words)

Example fix

# before
text = whisper.normalizers.EnglishTextNormalizer()(transcript)  # ValueError: Unexpected token

# after
from whisper.normalizers import EnglishTextNormalizer
norm = EnglishTextNormalizer()
try:
    text = norm(transcript)
except ValueError:
    text = transcript  # keep raw text for this segment

def normalize_safe(normalizer, text: str) -> str:
    try:
        return normalizer(text)
    except ValueError:
        return text  # fall back to raw ASR text

When it happens

Trigger: Calling EnglishTextNormalizer (used by default in transcribe()) on transcripts containing unusual spoken-number sequences, e.g. 'oh point oh', year-like compounds, or strings where num2words emitted a token the switch does not map; typically only for current=='NUM' style tokens that reach the final else.

Common situations: Normalizing ASR output at scale; certain input strings ('twenty twenty four point', mixed 'and a half' replacements from preprocess) hit uncovered branches on specific whisper/transformers versions; this exact bug has appeared and been patched across versions.

Understand the failure class

• Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

AI-assisted analysis of openai/whisper@5f86d1d863 (2026-08-14). Data as JSON: /api/errors/a680754ef371872c. Report an issue: GitHub.