sgl-project/sglang · error · ValueError

Cosmos3 policy input requires an observation image

Error message

Cosmos3 policy input requires an observation image

What it means

Cosmos3's policy mode predicts actions from an observation image; without one the request cannot be processed. The endpoint extracts images via _images_from_observation and raises if the list is empty when action_mode='policy'. Images can be supplied as URLs/paths or embedded image payloads in the observation.

Source

Thrown at python/sglang/multimodal_gen/runtime/entrypoints/action/cosmos3.py:174

            raise ValueError(
                "Cosmos3 requires num_frames == action_horizon + 1, got "
                f"num_frames={num_frames}, action_horizon={action_horizon}"
            )
        num_frames = expected_num_frames
    else:
        num_frames = int(num_frames)
        if num_frames <= 1:
            raise ValueError("Cosmos3 action num_frames must be greater than 1")
    if (num_frames - 1) % 4 != 0:
        raise ValueError(
            "Cosmos3 action_horizon must be divisible by 4 so num_frames "
            "is compatible with the temporal VAE"
        )

    images = _images_from_observation(observation)
    video_path = options.get("video_path") or observation.get("video")
    if action_mode == "policy" and not images:
        raise ValueError("Cosmos3 policy input requires an observation image")
    if action_mode == "inverse_dynamics" and video_path is None:
        raise ValueError("Cosmos3 inverse_dynamics input requires an observation video")
    if images and video_path is not None:
        raise ValueError("Cosmos3 action requests accept either an image or a video")
    batch_size = len(images) if images else 1
    max_batch_size = max(1, int(getattr(server_args, "batching_max_size", 1)))
    if batch_size > max_batch_size:
        raise ValueError(
            f"Cosmos3 action batch size {batch_size} exceeds "
            f"--batching-max-size={max_batch_size}"
        )
    image_path = None if not images else images[0] if batch_size == 1 else images

    domain_id = options.get("domain_id")
    domain_name = options.get("domain_name")
    raw_action_dim = options.get("raw_action_dim")
    if domain_id is None and not domain_name:
        raise ValueError("Cosmos3 action requests require domain_name or domain_id")

View on GitHub (pinned to 0132848349)

Solutions

1. Add an observation image (image_path/image_url or b64 image payload) to the request
2. Verify the image field name matches what _images_from_observation parses
3. If your observation is a video, use action_mode='inverse_dynamics' instead

Example fix

# before
{"observation": {"prompt": "pick cube"}}
# after
{"observation": {"prompt": "pick cube", "image": "</path/or/url>"}}

assert observation.get('image') or observation.get('image_url'), 'policy mode needs an observation image'

When it happens

Trigger: POST /v1/actions with action_mode='policy' (or default) but no image field, or an image field _images_from_observation doesn't recognize (empty string, empty list).

Common situations: Client sends only a video for a policy query; image key typo (e.g. 'img' instead of the expected image/image_url key); empty b64 payload; mixing up policy and inverse_dynamics request shapes.

Related errors

• Cosmos3 inverse_dynamics input requires an observation video
• Cosmos3 action requests accept either an image or a video
• unsupported input for causal Conv3D cat/pad CUDA
• unsupported input for usp_merge_heads CUDA
• unsupported input for modulate_scale_shift CUDA

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/c4233b9a596394e5. Report an issue: GitHub.