{"record":{"id":"9cc8079e33f8b8eb","repo":"sgl-project/sglang","slug":"request-mixes-standalone-audio-and-video-with-audi","errorCode":null,"errorMessage":"Request mixes standalone audio and video-with-audio; EPD merge path for this combination is not yet implemented.","messagePattern":"Request mixes standalone audio and video-with-audio; EPD merge path for this combination is not yet implemented\\.","errorType":"exception","errorClass":"NotImplementedError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/multimodal/processors/mimo_v2.py","lineNumber":2131,"sourceCode":"            for i, nu in enumerate(video_audio_per_video_num_units):\n                if nu <= 0:\n                    continue\n                seg_lens = list(video_audio_segment_lens_flat[off : off + nu])\n                off += nu\n                per_video_audio_info[i] = {\n                    \"segment_lens\": seg_lens,\n                    \"audio_token_len\": (\n                        int(video_audio_feature_lens[av_idx].item())\n                        if video_audio_feature_lens is not None\n                        else sum(seg_lens)\n                    ),\n                }\n                av_idx += 1\n\n        # Merge video-borne audio into AUDIO bucket for uniform slicing.\n        if video_audio_embedding is not None:\n            if Modality.AUDIO in embeddings:\n                raise NotImplementedError(\n                    \"Request mixes standalone audio and video-with-audio; \"\n                    \"EPD merge path for this combination is not yet implemented.\"\n                )\n            embeddings = dict(embeddings)\n            embeddings[Modality.AUDIO] = video_audio_embedding\n\n        merge_size = self.spatial_merge_size\n        input_ids = []\n        img_idx = video_idx = audio_idx = 0\n        for part in text_parts:\n            mod = self.mm_tokens.get_modality_of_token(part)\n            if mod == Modality.IMAGE:\n                grid = img_grid_thw[img_idx]\n                n = int(grid.prod().item()) // (merge_size**2)\n                input_ids += (\n                    [mp.vision_start_token_id]\n                    + [mp.image_token_id] * n\n                    + [mp.vision_end_token_id]","sourceCodeStart":2113,"sourceCodeEnd":2149,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/multimodal/processors/mimo_v2.py#L2113-L2149","documentation":"A NotImplementedError raised in get_mm_data when the request both includes standalone audio (Modality.AUDIO already in embeddings) and video that carries its own audio track (video_audio_embedding is not None). Merging the two audio sources into one slicing bucket is unimplemented in this version.","triggerScenarios":"Sending a single request containing an audio attachment plus a video-with-audio attachment to a MiMo-V2 EPD endpoint — the merge path for combining video-borne audio with standalone audio doesn't exist yet.","commonSituations":"Clients attaching a voice note and a video clip in the same message; multimodal chat UIs that bundle all attachments per turn; testing the EPD encoder with mixed modalities.","solutions":["Split into separate requests: one with the audio attachment, one with the video","Strip the audio track from the video (re-encode without audio) if the standalone audio is the important input","Watch sglang releases for the implemented merge path and upgrade"],"exampleFix":"# before (single request)\nmessages = [{'role':'user','content':[\n   {'type':'audio','audio': url_audio},\n   {'type':'video','video': url_video_with_sound},  # → NotImplementedError\n]}]\n# after: two requests\nreq1 = [{'role':'user','content':[{'type':'audio','audio': url_audio}, 'transcribe']}]\nreq2 = [{'role':'user','content':[{'type':'video','video': url_video_with_sound}, 'describe']}]","handlingStrategy":"validation","validationCode":"def has_mixed_audio(items):\n    standalone = any(i.get('type') == 'audio' for i in items)\n    video = any(i.get('type') == 'video' for i in items)\n    return standalone and video  # video may carry audio → not yet supported together\n\nif has_mixed_audio(content):\n    raise ClientSideError('send audio and video in separate requests')","typeGuard":null,"tryCatchPattern":"try:\n    emb = processor.get_mm_data(...)\nexcept NotImplementedError as e:\n    if 'standalone audio and video-with-audio' in str(e):\n        return error_response(400, 'split audio and video into separate requests')\n    raise","preventionTips":["UI: allow only one media category with audio per message","Strip audio tracks from videos when a standalone audio is also attached","Track sglang release notes for the merged EPD audio path"],"tags":["audio","video","not-implemented","epd","multimodal"],"backgroundTag":"unsupported-feature-combination","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}