{"record":{"id":"0a20e73418b07924","repo":"Comfy-Org/ComfyUI","slug":"minimax-music3-tokenizer-mismatch-for-token-exp","errorCode":null,"errorMessage":"MiniMax Music3 tokenizer mismatch for {token}: expected {expected}, got {token_id}","messagePattern":"MiniMax Music3 tokenizer mismatch for (.+?): expected (.+?), got (.+?)","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"comfy/ldm/minimax_music/prompt.py","lineNumber":70,"sourceCode":"    text = text.replace(\" ^ \", \"\\n\")\n    return f\"[start]\\n{text}\"\n\n\ndef build_prompt(caption, lyrics):\n    return (\n        \"<|im_start|><|caption_start|>\"\n        f\"{clean_caption(caption)}\"\n        \"<|caption_end|><|lyrics_start|>\"\n        f\"{normalize_lyrics(lyrics)}\"\n        \"<|lyrics_end|><|im_end|><|audio_start|>\"\n    )\n\n\ndef validate_tokenizer(tokenizer):\n    for token, expected in SPECIAL_TOKEN_IDS.items():\n        token_id = tokenizer.convert_tokens_to_ids(token)\n        if token_id != expected:\n            raise ValueError(f\"MiniMax Music3 tokenizer mismatch for {token}: expected {expected}, got {token_id}\")\n","sourceCodeStart":52,"sourceCodeEnd":71,"githubUrl":"https://github.com/Comfy-Org/ComfyUI/blob/1c6d8d45b3693bfbb32385b410d813a7fd6be216/comfy/ldm/minimax_music/prompt.py#L52-L71","documentation":"Raised by validate_tokenizer() when the loaded MiniMax Music3 tokenizer maps a special token (e.g. '<|audio_end|>', '<|caption_start|>') to an id different from the hardcoded SPECIAL_TOKEN_IDS table. The AR model's vocab masking, stop detection, and prompt template all use fixed ids, so a mismatched tokenizer silently corrupts generation; this check fails fast instead.","triggerScenarios":"Loading a tokenizer from the wrong checkpoint/revision, or a tokenizer.json whose added_tokens were renumbered, then calling validate_tokenizer (or generation, which relies on the same ids).","commonSituations":"Mixing tokenizer files from a different Music3 release; letting a hub library auto-upgrade the tokenizer; manually editing vocab files; partial download truncating the special-token table.","solutions":["Re-download the tokenizer files that shipped with the exact Music3 checkpoint you are loading","Compare tokenizer.convert_tokens_to_ids(t) for each special token against SPECIAL_TOKEN_IDS to see which ones drifted","Do not hand-edit vocab/added-token files; replace the whole tokenizer directory as a unit"],"exampleFix":"# before\nvalidate_tokenizer(tokenizer)  # raises\n# after: use the tokenizer bundled with the matching checkpoint\ntokenizer = load_tokenizer(checkpoint_dir / 'tokenizer')\nvalidate_tokenizer(tokenizer)","handlingStrategy":"validation","validationCode":"from comfy.ldm.minimax_music.prompt import validate_tokenizer\nvalidate_tokenizer(tokenizer)  # call once right after loading, fail fast with a clear message","typeGuard":null,"tryCatchPattern":"try:\n    validate_tokenizer(tokenizer)\nexcept ValueError as e:\n    raise RuntimeError(f'Wrong tokenizer for this checkpoint: {e}; re-download matching files') from e","preventionTips":["Always download tokenizer and model weights from the same release","Never edit vocab/added-token files by hand"],"tags":["minimax","music","tokenizer","model-loading"],"backgroundTag":null,"analyzedSha":"1c6d8d45b3693bfbb32385b410d813a7fd6be216","analyzedAt":"2026-08-14T19:37:18.893Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}