sgl-project/sglang · error · ValueError

Cosmos3 action requests accept either an image or a video

Error message

Cosmos3 action requests accept either an image or a video

What it means

A Cosmos3 action request must be either image-based or video-based, not both. The endpoint raises when _images_from_observation returns images AND a video_path is present, because the model input construction would be ambiguous. This catches payloads that accidentally include both media types.

Source

Thrown at python/sglang/multimodal_gen/runtime/entrypoints/action/cosmos3.py:178

        num_frames = expected_num_frames
    else:
        num_frames = int(num_frames)
        if num_frames <= 1:
            raise ValueError("Cosmos3 action num_frames must be greater than 1")
    if (num_frames - 1) % 4 != 0:
        raise ValueError(
            "Cosmos3 action_horizon must be divisible by 4 so num_frames "
            "is compatible with the temporal VAE"
        )

    images = _images_from_observation(observation)
    video_path = options.get("video_path") or observation.get("video")
    if action_mode == "policy" and not images:
        raise ValueError("Cosmos3 policy input requires an observation image")
    if action_mode == "inverse_dynamics" and video_path is None:
        raise ValueError("Cosmos3 inverse_dynamics input requires an observation video")
    if images and video_path is not None:
        raise ValueError("Cosmos3 action requests accept either an image or a video")
    batch_size = len(images) if images else 1
    max_batch_size = max(1, int(getattr(server_args, "batching_max_size", 1)))
    if batch_size > max_batch_size:
        raise ValueError(
            f"Cosmos3 action batch size {batch_size} exceeds "
            f"--batching-max-size={max_batch_size}"
        )
    image_path = None if not images else images[0] if batch_size == 1 else images

    domain_id = options.get("domain_id")
    domain_name = options.get("domain_name")
    raw_action_dim = options.get("raw_action_dim")
    if domain_id is None and not domain_name:
        raise ValueError("Cosmos3 action requests require domain_name or domain_id")
    if domain_id is not None and not domain_name and raw_action_dim is None:
        raise ValueError("raw_action_dim is required when only domain_id is provided")

    prompt = observation.get("prompt")

View on GitHub (pinned to 0132848349)

Solutions

1. Remove one of the two media fields — keep image for policy, video for inverse_dynamics
2. If fields may linger from a template, delete keys explicitly: payload['observation'].pop('video', None)

Example fix

# before
{"observation": {"image": "f0.png", "video": "rollout.mp4"}}
# after
{"observation": {"image": "f0.png"}}

has_img = bool(observation.get('image') or observation.get('image_url'))
has_vid = bool(params.get('video_path') or observation.get('video'))
assert not (has_img and has_vid), 'send image OR video, not both'

When it happens

Trigger: Sending an observation containing both an image and a video field in a single /v1/actions request.

Common situations: Reusing a generic multimodal template that always includes all fields; failed cleanup when switching a request from policy to inverse_dynamics; empty-string image fields that still parse as images.

Related errors

• Cosmos3 policy input requires an observation image
• Cosmos3 inverse_dynamics input requires an observation video
• unsupported input for causal Conv3D cat/pad CUDA
• unsupported input for usp_merge_heads CUDA
• unsupported input for modulate_scale_shift CUDA

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/6c204daadfbe026c. Report an issue: GitHub.