{"record":{"id":"5492273ccce8c19e","repo":"sgl-project/sglang","slug":"cosmos3-rollout-does-not-support-action-sound-moda","errorCode":null,"errorMessage":"Cosmos3 rollout does not support action/sound modalities.","messagePattern":"Cosmos3 rollout does not support action/sound modalities\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/multimodal_gen/runtime/pipelines_core/stages/model_specific_stages/cosmos3.py","lineNumber":1064,"sourceCode":"        uncond_text_ids = batch.extra[\"uncond_text_ids\"]\n        uncond_text_mask = batch.extra[\"uncond_text_mask\"]\n        video_shape = batch.extra[\"video_shape\"]\n        fps = batch.extra.get(\"fps\", 24.0)\n        velocity_mask = batch.extra.get(\"velocity_mask\")\n        condition_latents = batch.extra.get(\"condition_latents\")\n        guidance_interval = getattr(batch.sampling_params, \"guidance_interval\", None)\n\n        # Rollout requests carry a per-request scheduler bound by the timestep stage.\n        scheduler = batch.scheduler if batch.scheduler is not None else self.scheduler\n        if batch.rollout:\n            if velocity_mask is not None or condition_latents is not None:\n                raise ValueError(\n                    \"Cosmos3 rollout supports T2V/T2I only; I2V/V2V \"\n                    \"conditioned-frame re-blending breaks the Gaussian \"\n                    \"transition assumption of the SDE log-prob math.\"\n                )\n            if action_latents is not None or sound_latents is not None:\n                raise ValueError(\n                    \"Cosmos3 rollout does not support action/sound modalities.\"\n                )\n            self._maybe_prepare_rollout(batch)\n            self._maybe_init_denoising_env_collection(\n                batch=batch,\n                pipeline_config=server_args.pipeline_config,\n                image_kwargs={},\n                pos_cond_kwargs={\n                    \"text_ids\": cond_text_ids,\n                    \"text_mask\": cond_text_mask,\n                    \"fps\": fps,\n                },\n                neg_cond_kwargs={\n                    \"text_ids\": uncond_text_ids,\n                    \"text_mask\": uncond_text_mask,\n                    \"fps\": fps,\n                },\n                guidance=None,","sourceCodeStart":1046,"sourceCodeEnd":1082,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/multimodal_gen/runtime/pipelines_core/stages/model_specific_stages/cosmos3.py#L1046-L1082","documentation":"In rollout mode the Cosmos3 stage has no SDE log-prob math for action or sound latents, so any request carrying action_latents or sound_latents is rejected with a NotImplementedError-like ValueError. Only unconditional (optionally text-conditioned) video/image generation is supported for RL sampling.","triggerScenarios":"batch.rollout=True together with action_latents (action-conditioned dynamics) or sound_latents (audio-conditioned generation) populated on the batch.","commonSituations":"Reusing a world-model rollout config that enabled action conditioning for training-time generation; upgrading a pipeline where sound support was recently added but rollout paths weren't extended.","solutions":["Remove action/sound latents from rollout batches (generate plain T2V rollouts)","For action rollouts, run the non-rollout forward path and compute rewards externally instead of via SDE log-probs","Track upstream support: request/patch rollout log-prob math for action and sound modalities"],"exampleFix":"# before\nbatch.action_latents = prepare_actions(...)  # then batch.rollout = True\n# after\nbatch.action_latents = None\nbatch.rollout = True","handlingStrategy":"validation","validationCode":"if batch.rollout:\n    assert batch.action_latents is None and getattr(batch, \"sound_latents\", None) is None","typeGuard":"def rollout_batch_is_plain(batch) -> bool:\n    return all(getattr(batch, f, None) is None for f in (\"action_latents\", \"sound_latents\", \"condition_latents\", \"velocity_mask\"))","tryCatchPattern":null,"preventionTips":["Build rollout requests through a dedicated factory that never attaches action/sound latents","Watch release notes for action/sound rollout support before enabling those modalities in RL loops"],"tags":["cosmos3","rollout","action-latents","sound-latents","not-implemented"],"backgroundTag":"unsupported-operation-mode","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}