{"record":{"id":"187b8eb93fd6ccce","repo":"Comfy-Org/ComfyUI","slug":"minimax-music3-tokenizer-mismatch-for-token","errorCode":null,"errorMessage":"MiniMax Music3 tokenizer mismatch for {token}","messagePattern":"MiniMax Music3 tokenizer mismatch for (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"comfy/text_encoders/minimax_music.py","lineNumber":48,"sourceCode":"        \"merged_qkv\": \"{}model.layers.0.self_attn.qkv_proj.weight\".format(prefix) in state_dict,\n        \"merged_mlp\": \"{}model.layers.0.mlp.gate_up_proj.weight\".format(prefix) in state_dict,\n        \"decoder_merged_qkv\": \"{}model.audio_decoder.layers.0.self_attn.qkv_proj.weight\".format(prefix) in state_dict,\n        \"decoder_merged_mlp\": \"{}model.audio_decoder.layers.0.mlp.gate_up_proj.weight\".format(prefix) in state_dict,\n    }\n\n\nclass MiniMaxMusic3Tokenizer:\n    def __init__(self, embedding_directory=None, tokenizer_data={}):\n        tokenizer_json = tokenizer_data.get(\"tokenizer_json\")\n        if tokenizer_json is None:\n            raise ValueError(\"MiniMax Music3 text encoder checkpoint is missing tokenizer_json\")\n        if torch.is_tensor(tokenizer_json):\n            tokenizer_json = tokenizer_json.detach().cpu().numpy().tobytes()\n        self.tokenizer_json = tokenizer_json\n        self.tokenizer = Tokenizer.from_str(tokenizer_json.decode(\"utf-8\"))\n        for token, expected in SPECIAL_TOKEN_IDS.items():\n            if self.tokenizer.token_to_id(token) != expected:\n                raise ValueError(f\"MiniMax Music3 tokenizer mismatch for {token}\")\n\n    def tokenize_with_weights(self, text, return_word_ids=False, **kwargs):\n        prompt = build_prompt(text, kwargs.get(\"lyrics\", \"\"))\n        token_ids = self.tokenizer.encode(prompt, add_special_tokens=False).ids\n        return {\n            \"minimax_music3\": [[(token, 1.0) for token in token_ids]],\n            \"seed\": int(kwargs.get(\"seed\", 0)),\n            \"max_audio_frames\": int(kwargs.get(\"max_audio_frames\", MAX_AUDIO_FRAMES)),\n            \"cfg_scale\": float(kwargs.get(\"cfg_scale\", CFG_SCALE)),\n            \"top_k\": int(kwargs.get(\"top_k\", CFG_TOP_K)),\n        }\n\n    def state_dict(self):\n        return {\"tokenizer_json\": torch.frombuffer(bytearray(self.tokenizer_json), dtype=torch.uint8)}\n\n    def decode(self, token_ids, skip_special_tokens=True):\n        return self.tokenizer.decode(token_ids, skip_special_tokens=skip_special_tokens)\n","sourceCodeStart":30,"sourceCodeEnd":66,"githubUrl":"https://github.com/Comfy-Org/ComfyUI/blob/1c6d8d45b3693bfbb32385b410d813a7fd6be216/comfy/text_encoders/minimax_music.py#L30-L66","documentation":"After loading the embedded tokenizer JSON, MiniMaxMusic3Tokenizer validates that the special tokens (from SPECIAL_TOKEN_IDS, e.g. audio/lyrics boundary tokens) map to the exact IDs the model was trained with. If any token_to_id lookup disagrees, the vocabulary does not match the model's embedding table, so token IDs would index wrong rows and silently corrupt conditioning — hence a hard failure naming the offending token.","triggerScenarios":"tokenizer_data['tokenizer_json'] contains a tokenizer.json whose vocabulary differs from the one the MiniMax Music3 model was trained with (different merges, added tokens, or version), so token_to_id(token) != expected for a token in SPECIAL_TOKEN_IDS.","commonSituations":"User re-bundled the encoder with a tokenizer.json from a different revision of the MiniMax repo; mixing tokenizer files between Music3 variants; hand-edited vocab files.","solutions":["Use the exact tokenizer.json that ships with the same MiniMax Music3 release as the model weights","Re-bundle the encoder with bundle_te.py from the matching commit so tokenizer and weights stay paired","If maintaining a fork, diff the failing token's ID between tokenizer.json and SPECIAL_TOKEN_IDS and align them"],"exampleFix":"// before\n# bundled with tokenizer.json from a different Music3 revision\ntok = MiniMaxMusic3Tokenizer(tokenizer_data=sd)  # ValueError: mismatch for <token>\n\n# after\n# re-bundle with the tokenizer.json from the same release as the weights\nassert tok.tokenizer.token_to_id('[audio]') == EXPECTED_ID  # guard before use","handlingStrategy":"validation","validationCode":"from tokenizers import Tokenizer\nimport json\ntok = Tokenizer.from_str(json.loads(open('tokenizer.json').read()) if False else open('tokenizer.json').read())\nfor token, expected in SPECIAL_TOKEN_IDS.items():\n    if tok.token_to_id(token) != expected:\n        raise ValueError(f'tokenizer/weights revision mismatch at {token}')","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Pair tokenizer.json and model weights from the same release commit","Re-run the vocab check after any re-bundle"],"tags":["text-encoder","tokenizer","vocab-mismatch","minimax","music"],"backgroundTag":null,"analyzedSha":"1c6d8d45b3693bfbb32385b410d813a7fd6be216","analyzedAt":"2026-08-14T19:37:18.893Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}