Comfy-Org/ComfyUI · error · ValueError

MiniMax Music3 tokenizer mismatch for {token}: expected {exp

Error message

MiniMax Music3 tokenizer mismatch for {token}: expected {expected}, got {token_id}

What it means

Raised by validate_tokenizer() when the loaded MiniMax Music3 tokenizer maps a special token (e.g. '<|audio_end|>', '<|caption_start|>') to an id different from the hardcoded SPECIAL_TOKEN_IDS table. The AR model's vocab masking, stop detection, and prompt template all use fixed ids, so a mismatched tokenizer silently corrupts generation; this check fails fast instead.

Source

Thrown at comfy/ldm/minimax_music/prompt.py:70

    text = text.replace(" ^ ", "\n")
    return f"[start]\n{text}"


def build_prompt(caption, lyrics):
    return (
        "<|im_start|><|caption_start|>"
        f"{clean_caption(caption)}"
        "<|caption_end|><|lyrics_start|>"
        f"{normalize_lyrics(lyrics)}"
        "<|lyrics_end|><|im_end|><|audio_start|>"
    )


def validate_tokenizer(tokenizer):
    for token, expected in SPECIAL_TOKEN_IDS.items():
        token_id = tokenizer.convert_tokens_to_ids(token)
        if token_id != expected:
            raise ValueError(f"MiniMax Music3 tokenizer mismatch for {token}: expected {expected}, got {token_id}")

View on GitHub (pinned to 1c6d8d45b3)

Solutions

1. Re-download the tokenizer files that shipped with the exact Music3 checkpoint you are loading
2. Compare tokenizer.convert_tokens_to_ids(t) for each special token against SPECIAL_TOKEN_IDS to see which ones drifted
3. Do not hand-edit vocab/added-token files; replace the whole tokenizer directory as a unit

Example fix

# before
validate_tokenizer(tokenizer)  # raises
# after: use the tokenizer bundled with the matching checkpoint
tokenizer = load_tokenizer(checkpoint_dir / 'tokenizer')
validate_tokenizer(tokenizer)

from comfy.ldm.minimax_music.prompt import validate_tokenizer
validate_tokenizer(tokenizer)  # call once right after loading, fail fast with a clear message

try:
    validate_tokenizer(tokenizer)
except ValueError as e:
    raise RuntimeError(f'Wrong tokenizer for this checkpoint: {e}; re-download matching files') from e

When it happens

Trigger: Loading a tokenizer from the wrong checkpoint/revision, or a tokenizer.json whose added_tokens were renumbered, then calling validate_tokenizer (or generation, which relies on the same ids).

Common situations: Mixing tokenizer files from a different Music3 release; letting a hub library auto-upgrade the tokenizer; manually editing vocab files; partial download truncating the special-token table.

Related errors

• MiniMax Music3 prompt has {prompt_tokens} tokens; maximum is
• MiniMax Music3 text encoder checkpoint is missing tokenizer_
• MiniMax Music3 tokenizer mismatch for {token}
• MiniMax Music3 generated zero audio frames
• INVALID_TAG_FILTER

AI-assisted analysis of Comfy-Org/ComfyUI@1c6d8d45b3 (2026-08-14). Data as JSON: /api/errors/0a20e73418b07924. Report an issue: GitHub.