{"record":{"id":"a680754ef371872c","repo":"openai/whisper","slug":"unexpected-token-current","errorCode":null,"errorMessage":"Unexpected token: {current}","messagePattern":"Unexpected token: (.+?)","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"warning","filePath":"whisper/normalizers/english.py","lineNumber":380,"sourceCode":"                        if value is not None:\n                            yield output(value)\n                        yield output(current)\n                elif current == \"double\" or current == \"triple\":\n                    if next in self.ones or next in self.zeros:\n                        repeats = 2 if current == \"double\" else 3\n                        ones = self.ones.get(next, 0)\n                        value = str(value or \"\") + str(ones) * repeats\n                        skip = True\n                    else:\n                        if value is not None:\n                            yield output(value)\n                        yield output(current)\n                elif current == \"point\":\n                    if next in self.decimals or next_is_numeric:\n                        value = str(value or \"\") + \".\"\n                else:\n                    # should all have been covered at this point\n                    raise ValueError(f\"Unexpected token: {current}\")\n            else:\n                # all should have been covered at this point\n                raise ValueError(f\"Unexpected token: {current}\")\n\n        if value is not None:\n            yield output(value)\n\n    def preprocess(self, s: str):\n        # replace \"<number> and a half\" with \"<number> point five\"\n        results = []\n\n        segments = re.split(r\"\\band\\s+a\\s+half\\b\", s)\n        for i, segment in enumerate(segments):\n            if len(segment.strip()) == 0:\n                continue\n            if i == len(segments) - 1:\n                results.append(segment)\n            else:","sourceCodeStart":362,"sourceCodeEnd":398,"githubUrl":"https://github.com/openai/whisper/blob/5f86d1d86363843179951550570367b37c5d6f78/whisper/normalizers/english.py#L362-L398","documentation":"EnglishNumberNormalizer.postprocess_steps iterates over tokens produced by its own num2words-based tokenization of spoken numbers. The first 'Unexpected token' branch fires inside the numeric-word handling chain (current is a known numeric token class but falls through all sub-branches: not a units/teens/tens word covered, not 'point' with a decimal following). It signals a tokenizer-state edge case in transcribed ASR text rather than user misconfiguration.","triggerScenarios":"Calling EnglishTextNormalizer (used by default in transcribe()) on transcripts containing unusual spoken-number sequences, e.g. 'oh point oh', year-like compounds, or strings where num2words emitted a token the switch does not map; typically only for current=='NUM' style tokens that reach the final else.","commonSituations":"Normalizing ASR output at scale; certain input strings ('twenty twenty four point', mixed 'and a half' replacements from preprocess) hit uncovered branches on specific whisper/transformers versions; this exact bug has appeared and been patched across versions.","solutions":["Upgrade openai-whisper — the English normalizer has had multiple fixes for unexpected-token inputs","Wrap normalization in try/except and fall back to the raw/un-normalized text for that segment","Reproduce with EnglishTextNormalizer(s) directly to isolate the offending substring, then pre-clean it (e.g. strip stray number words)"],"exampleFix":"# before\ntext = whisper.normalizers.EnglishTextNormalizer()(transcript)  # ValueError: Unexpected token\n\n# after\nfrom whisper.normalizers import EnglishTextNormalizer\nnorm = EnglishTextNormalizer()\ntry:\n    text = norm(transcript)\nexcept ValueError:\n    text = transcript  # keep raw text for this segment","handlingStrategy":"fallback","validationCode":null,"typeGuard":null,"tryCatchPattern":"def normalize_safe(normalizer, text: str) -> str:\n    try:\n        return normalizer(text)\n    except ValueError:\n        return text  # fall back to raw ASR text","preventionTips":["Never let the English normalizer run un-caught over bulk corpora","Pin/upgrade openai-whisper to a release whose normalizer covers your number idioms","Unit-test the normalizer against your domain's spoken-number patterns (years, decimals, 'and a half')"],"tags":["normalization","english","asr-postprocessing","edge-case"],"backgroundTag":null,"analyzedSha":"5f86d1d86363843179951550570367b37c5d6f78","analyzedAt":"2026-08-14T18:53:59.547Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}