{"record":{"id":"fb9ad3d37728f446","repo":"sgl-project/sglang","slug":"pi05-action-state-broadcast-returned-none","errorCode":null,"errorMessage":"Pi05 action state broadcast returned None","messagePattern":"Pi05 action state broadcast returned None","errorType":"error_code","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"python/sglang/multimodal_gen/runtime/models/vlas/pi05_policy.py","lineNumber":1091,"sourceCode":"        }\n\n    def _broadcast_initial_action_state(\n        self,\n        x_t: torch.Tensor | None,\n    ) -> torch.Tensor:\n        split = get_vla_split_group()\n        if split is None:\n            if x_t is None:\n                raise RuntimeError(\"Pi05 action state is missing on single-rank run\")\n            return x_t\n        x_t = broadcast_tensor_from_rank(\n            x_t,\n            split,\n            src=split.action_root,\n            device=self.device,\n        )\n        if x_t is None:\n            raise RuntimeError(\"Pi05 action state broadcast returned None\")\n        return x_t\n\n    def _shard_action_sequence(self, x_t: torch.Tensor) -> tuple[torch.Tensor, int]:\n        sp_world_size = get_sequence_parallel_world_size()\n        sp_rank = get_sp_parallel_rank()\n        local_len = x_t.shape[1] // sp_world_size\n        start = sp_rank * local_len\n        end = start + local_len\n        return x_t[:, start:end].contiguous(), start\n\n    def sample_actions(\n        self,\n        observation: VLAObservationBatch,\n        prefix_context: PrefixContext,\n        *,\n        noise: torch.Tensor | None,\n        num_steps: int,\n        use_cuda_graph: bool = True,","sourceCodeStart":1073,"sourceCodeEnd":1109,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/multimodal_gen/runtime/models/vlas/pi05_policy.py#L1073-L1109","documentation":"In the multi-rank action sequence-parallel path, sample_actions broadcasts the initial action tensor from the action-root rank via broadcast_tensor_from_rank; if that helper returns None (e.g. non-root ranks had nothing to receive, a group misconfiguration, or a collective that silently failed), the policy refuses to continue with a None state. This is a defensive invariant on the distributed broadcast result.","triggerScenarios":"Running Pi05 with action sequence-parallelism where broadcast_tensor_from_rank returns None: mismatched world sizes, wrong src rank (action_root not part of the split group), CUDA/NCCL collective failure, or ranks taking divergent code paths so root never sent.","commonSituations":"Misconfigured TP/SP launch flags (e.g. --tp-size vs sequence-parallel size mismatch); one rank crashing or hitting a different branch; NCCL timeouts being swallowed; version skew between ranks or sglang nodes.","solutions":["Verify the distributed launch: world size, TP/SP settings, and that every rank enters sample_actions with the same action_sp_enabled flag","Ensure the action root rank actually holds/creates x_t before broadcast (check rank ordering logic)","Check NCCL logs for collective errors and set NCCL_DEBUG=INFO to diagnose","Reproduce single-rank to confirm the policy itself is healthy, then fix the distributed config"],"exampleFix":null,"handlingStrategy":"validation","validationCode":"from sglang.multimodal_gen.runtime.models.vlas.pi05_policy import get_vla_split_group\nsplit = get_vla_split_group()\nassert split is not None, \"VLA split group not initialized\"\nassert 0 <= split.action_root < dist.get_world_size(split.group), \"action_root out of range\"","typeGuard":null,"tryCatchPattern":"try:\n    actions = policy.sample_actions(prefix, suffix, x_t=x_t)\nexcept RuntimeError as e:\n    if \"broadcast returned None\" in str(e):\n        # collective misconfiguration: fail fast with context, do not retry blindly\n        raise RuntimeError(f\"action broadcast failed on rank {dist.get_rank()}; \"\n                           \"check SP/TP config and NCCL logs\") from e\n    raise","preventionTips":["Use one canonical launch command for all ranks; never mix TP/SP flag values across nodes","Set NCCL_DEBUG=INFO and watchdog timeouts during bring-up to surface collective failures","Smoke-test multi-rank inference with a tiny batch before production traffic"],"tags":["pi05","distributed","broadcast","sequence-parallel","nccl"],"backgroundTag":"distributed-collective-failure","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}