sgl-project/sglang · error · ValueError

Cosmos3 rollout supports T2V/T2I only; I2V/V2V conditioned-f

Error message

Cosmos3 rollout supports T2V/T2I only; I2V/V2V conditioned-frame re-blending breaks the Gaussian transition assumption of the SDE log-prob math.

What it means

During rollout (RL sampling) the Cosmos3 stage only supports text-to-video/text-to-image. Requests with velocity_mask or condition_latents (I2V/V2V) would require re-blending conditioned frames between SDE steps, which invalidates the Gaussian transition assumption used in the SDE log-probability computation, so the stage refuses them.

Source

Thrown at python/sglang/multimodal_gen/runtime/pipelines_core/stages/model_specific_stages/cosmos3.py:1058

        generator = batch.generator
        if generator is None and batch.seed is not None:
            generator = torch.Generator(device=latents.device).manual_seed(batch.seed)

        cond_text_ids = batch.extra["cond_text_ids"]
        cond_text_mask = batch.extra["cond_text_mask"]
        uncond_text_ids = batch.extra["uncond_text_ids"]
        uncond_text_mask = batch.extra["uncond_text_mask"]
        video_shape = batch.extra["video_shape"]
        fps = batch.extra.get("fps", 24.0)
        velocity_mask = batch.extra.get("velocity_mask")
        condition_latents = batch.extra.get("condition_latents")
        guidance_interval = getattr(batch.sampling_params, "guidance_interval", None)

        # Rollout requests carry a per-request scheduler bound by the timestep stage.
        scheduler = batch.scheduler if batch.scheduler is not None else self.scheduler
        if batch.rollout:
            if velocity_mask is not None or condition_latents is not None:
                raise ValueError(
                    "Cosmos3 rollout supports T2V/T2I only; I2V/V2V "
                    "conditioned-frame re-blending breaks the Gaussian "
                    "transition assumption of the SDE log-prob math."
                )
            if action_latents is not None or sound_latents is not None:
                raise ValueError(
                    "Cosmos3 rollout does not support action/sound modalities."
                )
            self._maybe_prepare_rollout(batch)
            self._maybe_init_denoising_env_collection(
                batch=batch,
                pipeline_config=server_args.pipeline_config,
                image_kwargs={},
                pos_cond_kwargs={
                    "text_ids": cond_text_ids,
                    "text_mask": cond_text_mask,
                    "fps": fps,
                },

View on GitHub (pinned to 0132848349)

Solutions

1. Filter rollout requests to T2V/T2I only (drop image/video conditioning inputs before setting rollout=True)
2. If I2V rollouts are required, use a different stage/model that supports conditioned-frame SDE log-probs
3. Gate in the request builder: assert condition inputs are None when rollout is requested

Example fix

# before
batch.rollout = True  # request also has condition_latents set
# after
assert batch.condition_latents is None and batch.velocity_mask is None, "T2V only for rollout"
batch.rollout = True

if batch.rollout:
    assert batch.condition_latents is None and getattr(batch, "velocity_mask", None) is None, "rollout is T2V/T2I only"

def is_rollout_safe(batch) -> bool:
    return batch.condition_latents is None and getattr(batch, "velocity_mask", None) is None

try:
    stage.forward(batch)
except ValueError as e:
    if "T2V/T2I only" in str(e):
        strip_conditioning(batch)  # or reroute request
    else:
        raise

When it happens

Trigger: Setting batch.rollout=True (rollout/sampling mode) on a request that also supplies condition_latents (image-conditioned video) or velocity_mask; typically triggered by an RL rollout engine feeding I2V/V2V tasks to the pipeline.

Common situations: Pointing an RL training loop's Cosmos3 rollout worker at an image-conditioned checkpoint or dataset; mixing T2V and I2V prompts in a rollout batch without filtering.

Related errors

• Cosmos3 accepts either --image-path (I2V) or --video-path (V
• Cosmos3 I2V image list is empty
• Cosmos3 rollout does not support action/sound modalities.
• rollout_sde_type must be one of {_VALID_ROLLOUT_SDE_TYPES},
• Cosmos3 action input accepts one image field; use a list or

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/40559d142d80148c. Report an issue: GitHub.