sgl-project/sglang · error · ValueError

Cosmos3 inverse_dynamics input requires an observation video

Error message

Cosmos3 inverse_dynamics input requires an observation video

What it means

Cosmos3's inverse_dynamics mode recovers the action between an initial frame and subsequent video frames, so it requires an observation video. The endpoint checks options.video_path or observation.video and raises if both are absent for action_mode='inverse_dynamics'.

Source

Thrown at python/sglang/multimodal_gen/runtime/entrypoints/action/cosmos3.py:176

                f"num_frames={num_frames}, action_horizon={action_horizon}"
            )
        num_frames = expected_num_frames
    else:
        num_frames = int(num_frames)
        if num_frames <= 1:
            raise ValueError("Cosmos3 action num_frames must be greater than 1")
    if (num_frames - 1) % 4 != 0:
        raise ValueError(
            "Cosmos3 action_horizon must be divisible by 4 so num_frames "
            "is compatible with the temporal VAE"
        )

    images = _images_from_observation(observation)
    video_path = options.get("video_path") or observation.get("video")
    if action_mode == "policy" and not images:
        raise ValueError("Cosmos3 policy input requires an observation image")
    if action_mode == "inverse_dynamics" and video_path is None:
        raise ValueError("Cosmos3 inverse_dynamics input requires an observation video")
    if images and video_path is not None:
        raise ValueError("Cosmos3 action requests accept either an image or a video")
    batch_size = len(images) if images else 1
    max_batch_size = max(1, int(getattr(server_args, "batching_max_size", 1)))
    if batch_size > max_batch_size:
        raise ValueError(
            f"Cosmos3 action batch size {batch_size} exceeds "
            f"--batching-max-size={max_batch_size}"
        )
    image_path = None if not images else images[0] if batch_size == 1 else images

    domain_id = options.get("domain_id")
    domain_name = options.get("domain_name")
    raw_action_dim = options.get("raw_action_dim")
    if domain_id is None and not domain_name:
        raise ValueError("Cosmos3 action requests require domain_name or domain_id")
    if domain_id is not None and not domain_name and raw_action_dim is None:
        raise ValueError("raw_action_dim is required when only domain_id is provided")

View on GitHub (pinned to 0132848349)

Solutions

1. Provide observation.video (path or URL) or parameters.video_path
2. If your observation is a single image, use action_mode='policy' instead

Example fix

# before
{"action_mode": "inverse_dynamics", "observation": {"image": "f0.png"}}
# after
{"action_mode": "inverse_dynamics", "observation": {"video": "rollout.mp4"}}

assert params.get('video_path') or observation.get('video'), 'inverse_dynamics mode needs a video'

When it happens

Trigger: POST /v1/actions with action_mode='inverse_dynamics' but no video/video_path in the request payload or observation.

Common situations: Sending an image where a video is required; video field name mismatch; upstream video upload failed silently leaving the field empty.

Related errors

• Cosmos3 policy input requires an observation image
• Cosmos3 action requests accept either an image or a video
• Cosmos3 inverse_dynamics prompt must be a string
• unsupported input for causal Conv3D cat/pad CUDA
• unsupported input for usp_merge_heads CUDA

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/9bdb072b56eff2a4. Report an issue: GitHub.