sgl-project/sglang · error · ValueError

video must be a tuple of (video_tensor, timestamps), but got

Error message

video must be a tuple of (video_tensor, timestamps), but got {type(video)}. Video download and decoding should be done by sglang load_video before calling process_video.

What it means

Raised by process_video when video_input.video is not a tuple of (video_tensor, timestamps). The MiMo-V2 pipeline expects decoding to have already been performed by sglang's load_video; passing raw bytes, a path, or a decord/HF video object directly violates that contract.

Source

Thrown at python/sglang/srt/multimodal/processors/mimo_v2.py:735

                if len(left_indices) > 0:
                    selected_frame_indices = left_indices[-1:].clone()
                else:
                    raise ValueError(
                        f"No frames before start_time {start_time} in all_timestamps {all_timestamps.tolist()}"
                    )
            else:
                selected_frame_indices = candidate_indices

            assert (
                len(selected_frame_indices) > 0
            ), f"No frames selected for segment {start_time} - {end_time} in all_timestamps {all_timestamps.tolist()}"
            return selected_frame_indices

        kwargs = self.prepare_video_kwargs(video_input)
        video = video_input.video

        if not isinstance(video, tuple):
            raise ValueError(
                f"video must be a tuple of (video_tensor, timestamps), but got {type(video)}. "
                "Video download and decoding should be done by sglang load_video before calling process_video."
            )

        video_tensor, timestamps_sampled = video
        if len(timestamps_sampled) < 2:
            logger.info(
                "[Warning] Less than two frames are sampled, using default fps (1 fps)"
            )
            fps_sampled = 1
        else:
            fps_sampled = 1 / (timestamps_sampled[1] - timestamps_sampled[0])
        num_frames_sampled = video_tensor.shape[0]

        start_time = (
            video_input.start_time
            if video_input.start_time is not None
            else timestamps_sampled[0]

View on GitHub (pinned to 0132848349)

Solutions

1. Run the video through sglang's load_video (the _preprocess_video_sync path) before process_video so .video is the (video_tensor, timestamps) tuple
2. If constructing VideoInput manually, decode first and set video=(frames_tensor, timestamps_tensor)
3. Align with the sglang version's expected internal representation after upgrades

Example fix

# before
vi = VideoInput(video=b'raw mp4 bytes'); proc.process_video(vi)  # ValueError
# after
frames, ts = load_video('clip.mp4', **sampling_kwargs)  # via sglang loader
vi = VideoInput(video=(frames, ts))
out = proc.process_video(vi)

assert isinstance(video_input.video, tuple) and len(video_input.video) == 2, \
    'decode via sglang load_video first; .video must be (tensor, timestamps)'

def is_decoded_video(v) -> bool:
    return (isinstance(v, tuple) and len(v) == 2
            and torch.is_tensor(v[0]) and torch.is_tensor(v[1]))

try:
    out = proc.process_video(video_input)
except ValueError as e:
    if 'video must be a tuple' in str(e):
        video_input.video = load_video(source, **sampling_kwargs)  # decode then retry
        out = proc.process_video(video_input)
    else:
        raise

When it happens

Trigger: Calling process_video with a VideoInput whose .video field holds undecoded data (path string, bytes, URL, or a HF video object) instead of the (tensor, timestamps) tuple produced by load_video.

Common situations: Bypassing the standard mm_data loading path in custom integrations; upgrading sglang versions where the internal video representation changed from raw to (tensor, timestamps); test code constructing VideoInput by hand.

Understand the failure class

Background: "Wrong argument type", "must be a string", "expected Array or Prism::Scope": TypeError and ArgumentError when a library receives a value of the wrong type — this error's family across 28 libraries.

Related errors

• bad metadata: dur={duration} h={original_height} w={original
• Unsupported video input type for EPD encoder: {type(video_da
• spt must be a bool when provided
• Unknown type: {type(other)}
• Invalid value: {other}

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/d92a518da44e8cba. Report an issue: GitHub.