{"record":{"id":"9cdd0d602bcaccb8","repo":"microsoft/VibeVoice","slug":"unsupported-modality-modality","errorCode":null,"errorMessage":"Unsupported modality: {modality}","messagePattern":"Unsupported modality: (.+?)","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"vllm_plugin/model.py","lineNumber":948,"sourceCode":"    \"\"\"\n    VibeVoice ASR model with native vLLM multimodal integration.\n    \n    This model combines VibeVoice acoustic/semantic tokenizers for audio encoding\n    with a causal language model for text generation.\n    \"\"\"\n    \n    @classmethod\n    def get_placeholder_str(cls, modality: str, i: int) -> str | None:\n        \"\"\"Return the placeholder string format for a given modality.\n        \n        Returns \"<|AUDIO|>\" which vLLM inserts into the conversation prompt.\n        This single placeholder is later expanded by `_get_prompt_updates` into:\n            [speech_start_id] + [speech_pad_id] * N + [speech_end_id] + [newline_id]\n        where N = ceil(audio_samples / compress_ratio).\n        \"\"\"\n        if modality.startswith(\"audio\"):\n            return \"<|AUDIO|>\"\n        raise ValueError(f\"Unsupported modality: {modality}\")\n    \n    def __init__(self, *, vllm_config: VllmConfig, prefix: str = \"\"):\n        super().__init__()\n        config = vllm_config.model_config.hf_config\n        self.config = config\n        \n        self.audio_encoder = VibeVoiceAudioEncoder(config)\n        \n        # Pass decoder_config to the language model initialization\n        decoder_config = getattr(config, \"decoder_config\", config)\n        self.language_model = init_vllm_registered_model(\n            vllm_config=vllm_config,\n            hf_config=decoder_config,\n            prefix=maybe_prefix(prefix, \"language_model\"),\n            architectures=[\"Qwen2ForCausalLM\"],\n        )\n        \n        self.make_empty_intermediate_tensors = (","sourceCodeStart":930,"sourceCodeEnd":966,"githubUrl":"https://github.com/microsoft/VibeVoice/blob/94da20d98b2fa7688e9cbfaf7692ddb4954f7600/vllm_plugin/model.py#L930-L966","documentation":"Raised by the plugin's static get_placeholder_str(modality, i) hook: vLLM asks each multimodal plugin which placeholder token to insert per modality, and this model only understands modalities whose name starts with \"audio\" (returns \"<|AUDIO|>\"). Any other modality string (image, video, etc.) is rejected.","triggerScenarios":"A chat request containing image or video content blocks routed to a VibeVoice-only model; another plugin registered in the same process calling get_placeholder_str with its own modality name; explicit calls like get_placeholder_str(\"image\", 0) in custom integration code.","commonSituations":"Reusing a multi-modal client pipeline (built for vision-language models) against an audio-only deployment; content-type autodetection labeling the payload as \"video\" for .mp4 files even though only the audio track is wanted.","solutions":["Send only audio content to this model; strip image/video blocks from the messages before calling the API.","For .mp4/.mkv input, extract the audio track first (ffmpeg -i in.mp4 -vn -ac 1 -ar 24000 out.wav) and submit that.","If you must serve mixed modalities, run a separate vision model and route per modality instead of relying on this plugin to accept them.","Confirm your modality key is literally 'audio' (not 'Audio', 'sound', 'speech')."],"exampleFix":"# before\nmessages = [{\"role\": \"user\", \"content\": [\n    {\"type\": \"image\", \"image\": img},\n    {\"type\": \"audio\", \"audio\": wav}] }]\n\n# after\nmessages = [{\"role\": \"user\", \"content\": [\n    {\"type\": \"audio\", \"audio\": wav}] }]","handlingStrategy":"type-guard","validationCode":"ALLOWED = {\"audio\"}\n\ndef filter_messages(messages: list) -> list:\n    \"\"\"Keep only content blocks this model can handle.\"\"\"\n    out = []\n    for m in messages:\n        content = m.get(\"content\")\n        if isinstance(content, list):\n            kept = [b for b in content if b.get(\"type\", \"\").startswith(\"audio\")]\n            if not kept:\n                raise ValueError(\"message has no audio content for VibeVoice\")\n            out.append({**m, \"content\": kept})\n        else:\n            out.append(m)\n    return out","typeGuard":"def is_supported_modality(modality: str) -> bool:\n    return isinstance(modality, str) and modality.startswith(\"audio\")","tryCatchPattern":null,"preventionTips":["Route requests by modality at the gateway: audio -> VibeVoice, image/video -> a vision model.","Extract audio tracks from video files (ffmpeg -vn) rather than sending the container.","Use the literal modality key 'audio' in all request payloads."],"tags":["modality","multimodal","prompt-template","vllm","routing"],"backgroundTag":null,"analyzedSha":"94da20d98b2fa7688e9cbfaf7692ddb4954f7600","analyzedAt":"2026-08-15T04:12:07.418Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}