{"record":{"id":"d5b378bc6a7f5dfa","repo":"sgl-project/sglang","slug":"tokenizer-missing-required-special-token-name-r","errorCode":null,"errorMessage":"tokenizer missing required special token {name!r}; checkpoint vocab does not match MiMo-V2-ASR","messagePattern":"tokenizer missing required special token (.+?); checkpoint vocab does not match MiMo-V2-ASR","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"critical","filePath":"python/sglang/srt/multimodal/processors/mimo_v2_asr.py","lineNumber":86,"sourceCode":"        self.mm_tokens = MultimodalSpecialTokens(\n            audio_token=f\"{self.AUDIO_START_TOKEN}{self.AUDIO_PAD_TOKEN}{self.AUDIO_END_TOKEN}\",\n            audio_token_id=self.audio_token_id,\n            audio_token_regex=self.AUDIO_REGEX,\n        ).build(_processor)\n\n    def __getattr__(self, name):\n        # Delegate audio_pipeline fields so callers can use self.audio_token_id\n        # etc. directly. Only triggers when normal attribute lookup fails;\n        # __dict__.get avoids recursion before audio_pipeline is assigned.\n        pipeline = self.__dict__.get(\"audio_pipeline\")\n        if pipeline is not None and hasattr(pipeline, name):\n            return getattr(pipeline, name)\n        raise AttributeError(name)\n\n    def _resolve_special_token_id(self, name: str) -> int:\n        tid = self.tokenizer.convert_tokens_to_ids(name)\n        if tid is None or tid == self.tokenizer.unk_token_id:\n            raise ValueError(\n                f\"tokenizer missing required special token {name!r}; \"\n                \"checkpoint vocab does not match MiMo-V2-ASR\"\n            )\n        return int(tid)\n\n    def _process_contents(self, contents: List[_Content]):\n        \"\"\"Run pipeline + tokenizer over an interleaved content list.\n\n        Returns ``(input_ids: Tensor[L], audio_inputs: list[Tensor],\n        position_ids: Tensor[3,L], rope_deltas: Tensor[1,1])``.\n        \"\"\"\n        input_ids: List[int] = []\n        audio_inputs: List[torch.Tensor] = []\n\n        for content in contents:\n            if content.type == \"text\":\n                if isinstance(content.content, str):\n                    input_ids.extend(self.tokenizer.encode(content.content))","sourceCodeStart":68,"sourceCodeEnd":104,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/multimodal/processors/mimo_v2_asr.py#L68-L104","documentation":"Raised by _resolve_special_token_id in the MiMo-V2-ASR pipeline when the tokenizer's convert_tokens_to_ids returns None or the unk token for a required special token. This means the loaded checkpoint's vocabulary doesn't contain the ASR-specific special tokens, i.e. you're pointing MiMo-V2-ASR code at a non-ASR (base) checkpoint.","triggerScenarios":"Initializing the MiMo-V2-ASR processor with a tokenizer from the base MiMo-V2 model (or a mismatched revision) so tokens like the ASR begin/end specials resolve to unk; also when tokenizer.json is from an older vocab version.","commonSituations":"Downloading the base model repo instead of the -ASR variant; reusing a cached tokenizer from a different model revision; vocab updates in the model family between releases.","solutions":["Use the correct MiMo-V2-ASR checkpoint and its bundled tokenizer files","Clear the HF cache for the mismatched revision and re-download the ASR repo","Verify: tokenizer.convert_tokens_to_ids('<asr_token>') should not be None/unk before init"],"exampleFix":"# before\nmodel = 'org/MiMo-V2'            # base vocab → ValueError\n# after\nmodel = 'org/MiMo-V2-ASR'        # checkpoint whose vocab contains ASR specials","handlingStrategy":"validation","validationCode":"tok = tokenizer\nfor name in REQUIRED_SPECIALS:  # e.g. ['<|asr_start|>', '<|asr_end|>', ...]\n    tid = tok.convert_tokens_to_ids(name)\n    assert tid is not None and tid != tok.unk_token_id, \\\n        f'checkpoint vocab lacks {name}; use the MiMo-V2-ASR checkpoint'","typeGuard":null,"tryCatchPattern":"try:\n    pipeline = MiMoV2ASRPipeline(model_path, tokenizer)\nexcept ValueError as e:\n    if 'checkpoint vocab does not match MiMo-V2-ASR' in str(e):\n        raise ConfigError('point --model-path at the MiMo-V2-ASR repo, not the base model')\n    raise","preventionTips":["Always pull tokenizer + weights from the same ASR repo revision","Clear stale HF cache entries when switching model variants","Add a vocab smoke test for required special tokens at startup"],"tags":["tokenizer","checkpoint-mismatch","vocab","asr","initialization"],"backgroundTag":"tokenizer-vocab-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T11:17:15.048Z"}