sgl-project/sglang · error · NotImplementedError

Request mixes standalone audio and video-with-audio; EPD mer

Error message

Request mixes standalone audio and video-with-audio; EPD merge path for this combination is not yet implemented.

What it means

A NotImplementedError raised in get_mm_data when the request both includes standalone audio (Modality.AUDIO already in embeddings) and video that carries its own audio track (video_audio_embedding is not None). Merging the two audio sources into one slicing bucket is unimplemented in this version.

Source

Thrown at python/sglang/srt/multimodal/processors/mimo_v2.py:2131

            for i, nu in enumerate(video_audio_per_video_num_units):
                if nu <= 0:
                    continue
                seg_lens = list(video_audio_segment_lens_flat[off : off + nu])
                off += nu
                per_video_audio_info[i] = {
                    "segment_lens": seg_lens,
                    "audio_token_len": (
                        int(video_audio_feature_lens[av_idx].item())
                        if video_audio_feature_lens is not None
                        else sum(seg_lens)
                    ),
                }
                av_idx += 1

        # Merge video-borne audio into AUDIO bucket for uniform slicing.
        if video_audio_embedding is not None:
            if Modality.AUDIO in embeddings:
                raise NotImplementedError(
                    "Request mixes standalone audio and video-with-audio; "
                    "EPD merge path for this combination is not yet implemented."
                )
            embeddings = dict(embeddings)
            embeddings[Modality.AUDIO] = video_audio_embedding

        merge_size = self.spatial_merge_size
        input_ids = []
        img_idx = video_idx = audio_idx = 0
        for part in text_parts:
            mod = self.mm_tokens.get_modality_of_token(part)
            if mod == Modality.IMAGE:
                grid = img_grid_thw[img_idx]
                n = int(grid.prod().item()) // (merge_size**2)
                input_ids += (
                    [mp.vision_start_token_id]
                    + [mp.image_token_id] * n
                    + [mp.vision_end_token_id]

View on GitHub (pinned to 0132848349)

Solutions

1. Split into separate requests: one with the audio attachment, one with the video
2. Strip the audio track from the video (re-encode without audio) if the standalone audio is the important input
3. Watch sglang releases for the implemented merge path and upgrade

Example fix

# before (single request)
messages = [{'role':'user','content':[
   {'type':'audio','audio': url_audio},
   {'type':'video','video': url_video_with_sound},  # → NotImplementedError
]}]
# after: two requests
req1 = [{'role':'user','content':[{'type':'audio','audio': url_audio}, 'transcribe']}]
req2 = [{'role':'user','content':[{'type':'video','video': url_video_with_sound}, 'describe']}]

def has_mixed_audio(items):
    standalone = any(i.get('type') == 'audio' for i in items)
    video = any(i.get('type') == 'video' for i in items)
    return standalone and video  # video may carry audio → not yet supported together

if has_mixed_audio(content):
    raise ClientSideError('send audio and video in separate requests')

try:
    emb = processor.get_mm_data(...)
except NotImplementedError as e:
    if 'standalone audio and video-with-audio' in str(e):
        return error_response(400, 'split audio and video into separate requests')
    raise

When it happens

Trigger: Sending a single request containing an audio attachment plus a video-with-audio attachment to a MiMo-V2 EPD endpoint — the merge path for combining video-borne audio with standalone audio doesn't exist yet.

Common situations: Clients attaching a voice note and a video clip in the same message; multimodal chat UIs that bundle all attachments per turn; testing the EPD encoder with mixed modalities.

Related errors

• Unsupported dims: {self.dims}
• gRPC encode only supports IMAGE modality, got: {non_image}
• {error_msg}
• MM inputs where only some items are precomputed.
• projection_cls = {projection_cls}, not implemented

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/9cc8079e33f8b8eb. Report an issue: GitHub.