microsoft/VibeVoice · error · ValueError

Unsupported modality: {modality}

Error message

Unsupported modality: {modality}

What it means

Raised by the plugin's static get_placeholder_str(modality, i) hook: vLLM asks each multimodal plugin which placeholder token to insert per modality, and this model only understands modalities whose name starts with "audio" (returns "<|AUDIO|>"). Any other modality string (image, video, etc.) is rejected.

Source

Thrown at vllm_plugin/model.py:948

    """
    VibeVoice ASR model with native vLLM multimodal integration.
    
    This model combines VibeVoice acoustic/semantic tokenizers for audio encoding
    with a causal language model for text generation.
    """
    
    @classmethod
    def get_placeholder_str(cls, modality: str, i: int) -> str | None:
        """Return the placeholder string format for a given modality.
        
        Returns "<|AUDIO|>" which vLLM inserts into the conversation prompt.
        This single placeholder is later expanded by `_get_prompt_updates` into:
            [speech_start_id] + [speech_pad_id] * N + [speech_end_id] + [newline_id]
        where N = ceil(audio_samples / compress_ratio).
        """
        if modality.startswith("audio"):
            return "<|AUDIO|>"
        raise ValueError(f"Unsupported modality: {modality}")
    
    def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""):
        super().__init__()
        config = vllm_config.model_config.hf_config
        self.config = config
        
        self.audio_encoder = VibeVoiceAudioEncoder(config)
        
        # Pass decoder_config to the language model initialization
        decoder_config = getattr(config, "decoder_config", config)
        self.language_model = init_vllm_registered_model(
            vllm_config=vllm_config,
            hf_config=decoder_config,
            prefix=maybe_prefix(prefix, "language_model"),
            architectures=["Qwen2ForCausalLM"],
        )
        
        self.make_empty_intermediate_tensors = (

View on GitHub (pinned to 94da20d98b)

Solutions

1. Send only audio content to this model; strip image/video blocks from the messages before calling the API.
2. For .mp4/.mkv input, extract the audio track first (ffmpeg -i in.mp4 -vn -ac 1 -ar 24000 out.wav) and submit that.
3. If you must serve mixed modalities, run a separate vision model and route per modality instead of relying on this plugin to accept them.
4. Confirm your modality key is literally 'audio' (not 'Audio', 'sound', 'speech').

Example fix

# before
messages = [{"role": "user", "content": [
    {"type": "image", "image": img},
    {"type": "audio", "audio": wav}] }]

# after
messages = [{"role": "user", "content": [
    {"type": "audio", "audio": wav}] }]

ALLOWED = {"audio"}

def filter_messages(messages: list) -> list:
    """Keep only content blocks this model can handle."""
    out = []
    for m in messages:
        content = m.get("content")
        if isinstance(content, list):
            kept = [b for b in content if b.get("type", "").startswith("audio")]
            if not kept:
                raise ValueError("message has no audio content for VibeVoice")
            out.append({**m, "content": kept})
        else:
            out.append(m)
    return out

def is_supported_modality(modality: str) -> bool:
    return isinstance(modality, str) and modality.startswith("audio")

When it happens

Trigger: A chat request containing image or video content blocks routed to a VibeVoice-only model; another plugin registered in the same process calling get_placeholder_str with its own modality name; explicit calls like get_placeholder_str("image", 0) in custom integration code.

Common situations: Reusing a multi-modal client pipeline (built for vision-language models) against an audio-only deployment; content-type autodetection labeling the payload as "video" for .mp4 files even though only the audio track is wanted.

Related errors

• Unsupported audio data type: {type(data)}
• Audio duration ({duration_sec:.1f}s) exceeds the configured
• Missing acoustic/semantic tokenizer config in model config
• acoustic_tokenizer_config has unexpected type: {type(ac_cfg)
• semantic_tokenizer_config has unexpected type: {type(sc_cfg)

AI-assisted analysis of microsoft/VibeVoice@94da20d98b (2026-08-15). Data as JSON: /api/errors/9cdd0d602bcaccb8. Report an issue: GitHub.