sgl-project/sglang · error · ValueError

Cosmos3 inverse_dynamics prompt must be a string

Error message

Cosmos3 inverse_dynamics prompt must be a string

What it means

In Cosmos3 inverse_dynamics mode the prompt (the described action/task) must be a plain string, not a list. Policy mode allows a list of per-image prompts, but inverse_dynamics processes a single video so a list prompt is rejected via isinstance(prompt, str).

Source

Thrown at python/sglang/multimodal_gen/runtime/entrypoints/action/cosmos3.py:200

        raise ValueError(
            f"Cosmos3 action batch size {batch_size} exceeds "
            f"--batching-max-size={max_batch_size}"
        )
    image_path = None if not images else images[0] if batch_size == 1 else images

    domain_id = options.get("domain_id")
    domain_name = options.get("domain_name")
    raw_action_dim = options.get("raw_action_dim")
    if domain_id is None and not domain_name:
        raise ValueError("Cosmos3 action requests require domain_name or domain_id")
    if domain_id is not None and not domain_name and raw_action_dim is None:
        raise ValueError("raw_action_dim is required when only domain_id is provided")

    prompt = observation.get("prompt")
    if prompt is None:
        prompt = observation.get("task", "")
    if action_mode != "policy" and not isinstance(prompt, str):
        raise ValueError("Cosmos3 inverse_dynamics prompt must be a string")
    prompt = _action_prompt(prompt, batch_size)
    sampling_kwargs = {
        "request_id": payload.get("request_id") or payload.get("id"),
        "prompt": prompt,
        "image_path": image_path,
        "video_path": video_path,
        "action_mode": action_mode,
        "domain_id": domain_id,
        "domain_name": domain_name,
        "raw_action_dim": raw_action_dim,
        "action_fps": options.get("action_fps"),
        "action_view_point": options.get("action_view_point", "ego_view"),
        "action_normalization": options.get("action_normalization", "quantile"),
        "action_stats_path": server_args.pipeline_config.action_stats_path,
        "num_frames": num_frames,
        "fps": int(options.get("fps", 5)),
        "height": int(options.get("height", 480)),
        "width": int(options.get("width", 832)),

View on GitHub (pinned to 0132848349)

Solutions

1. Pass prompt as a single string for inverse_dynamics requests
2. If reusing policy code paths, take prompt[0] or join the list when switching modes

Example fix

# before
{"action_mode": "inverse_dynamics", "observation": {"prompt": ["lift arm"]}}
# after
{"action_mode": "inverse_dynamics", "observation": {"prompt": "lift arm"}}

if mode == 'inverse_dynamics':
    assert isinstance(observation.get('prompt', observation.get('task','')), str)

def is_inverse_prompt_ok(mode, prompt) -> bool:
    return mode == 'policy' or isinstance(prompt, str)

When it happens

Trigger: POST /v1/actions with action_mode='inverse_dynamics' and observation.prompt = ["a", "b"] (or a non-string like an int after the task fallback).

Common situations: Reusing a batched policy client payload for inverse_dynamics; prompt falling back to observation.task containing structured data.

Related errors

• Cosmos3 action prompt list must contain only strings
• Cosmos3 batched action input requires one prompt per image,
• Cosmos3 inverse_dynamics input requires an observation video
• num_token_non_padded must be a torch.Tensor
• actions must be a list[list[str]]

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/66a0c50be9a5509f. Report an issue: GitHub.