sgl-project/sglang · error · ValueError

Cosmos3 rollout does not support action/sound modalities.

Error message

Cosmos3 rollout does not support action/sound modalities.

What it means

In rollout mode the Cosmos3 stage has no SDE log-prob math for action or sound latents, so any request carrying action_latents or sound_latents is rejected with a NotImplementedError-like ValueError. Only unconditional (optionally text-conditioned) video/image generation is supported for RL sampling.

Source

Thrown at python/sglang/multimodal_gen/runtime/pipelines_core/stages/model_specific_stages/cosmos3.py:1064

        uncond_text_ids = batch.extra["uncond_text_ids"]
        uncond_text_mask = batch.extra["uncond_text_mask"]
        video_shape = batch.extra["video_shape"]
        fps = batch.extra.get("fps", 24.0)
        velocity_mask = batch.extra.get("velocity_mask")
        condition_latents = batch.extra.get("condition_latents")
        guidance_interval = getattr(batch.sampling_params, "guidance_interval", None)

        # Rollout requests carry a per-request scheduler bound by the timestep stage.
        scheduler = batch.scheduler if batch.scheduler is not None else self.scheduler
        if batch.rollout:
            if velocity_mask is not None or condition_latents is not None:
                raise ValueError(
                    "Cosmos3 rollout supports T2V/T2I only; I2V/V2V "
                    "conditioned-frame re-blending breaks the Gaussian "
                    "transition assumption of the SDE log-prob math."
                )
            if action_latents is not None or sound_latents is not None:
                raise ValueError(
                    "Cosmos3 rollout does not support action/sound modalities."
                )
            self._maybe_prepare_rollout(batch)
            self._maybe_init_denoising_env_collection(
                batch=batch,
                pipeline_config=server_args.pipeline_config,
                image_kwargs={},
                pos_cond_kwargs={
                    "text_ids": cond_text_ids,
                    "text_mask": cond_text_mask,
                    "fps": fps,
                },
                neg_cond_kwargs={
                    "text_ids": uncond_text_ids,
                    "text_mask": uncond_text_mask,
                    "fps": fps,
                },
                guidance=None,

View on GitHub (pinned to 0132848349)

Solutions

1. Remove action/sound latents from rollout batches (generate plain T2V rollouts)
2. For action rollouts, run the non-rollout forward path and compute rewards externally instead of via SDE log-probs
3. Track upstream support: request/patch rollout log-prob math for action and sound modalities

Example fix

# before
batch.action_latents = prepare_actions(...)  # then batch.rollout = True
# after
batch.action_latents = None
batch.rollout = True

if batch.rollout:
    assert batch.action_latents is None and getattr(batch, "sound_latents", None) is None

def rollout_batch_is_plain(batch) -> bool:
    return all(getattr(batch, f, None) is None for f in ("action_latents", "sound_latents", "condition_latents", "velocity_mask"))

When it happens

Trigger: batch.rollout=True together with action_latents (action-conditioned dynamics) or sound_latents (audio-conditioned generation) populated on the batch.

Common situations: Reusing a world-model rollout config that enabled action conditioning for training-time generation; upgrading a pipeline where sound support was recently added but rollout paths weren't extended.

Related errors

• Cosmos3 rollout supports T2V/T2I only; I2V/V2V conditioned-f
• Cosmos3 action generation does not support CFG parallel yet
• /v1/models ${response.status}
• Unsupported msgpack byte ${b}
• This browser does not support gzip stream decoding

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/5492273ccce8c19e. Report an issue: GitHub.