sgl-project/sglang · error · RuntimeError

Pi05 action state broadcast returned None

Error message

Pi05 action state broadcast returned None

What it means

In the multi-rank action sequence-parallel path, sample_actions broadcasts the initial action tensor from the action-root rank via broadcast_tensor_from_rank; if that helper returns None (e.g. non-root ranks had nothing to receive, a group misconfiguration, or a collective that silently failed), the policy refuses to continue with a None state. This is a defensive invariant on the distributed broadcast result.

Source

Thrown at python/sglang/multimodal_gen/runtime/models/vlas/pi05_policy.py:1091

        }

    def _broadcast_initial_action_state(
        self,
        x_t: torch.Tensor | None,
    ) -> torch.Tensor:
        split = get_vla_split_group()
        if split is None:
            if x_t is None:
                raise RuntimeError("Pi05 action state is missing on single-rank run")
            return x_t
        x_t = broadcast_tensor_from_rank(
            x_t,
            split,
            src=split.action_root,
            device=self.device,
        )
        if x_t is None:
            raise RuntimeError("Pi05 action state broadcast returned None")
        return x_t

    def _shard_action_sequence(self, x_t: torch.Tensor) -> tuple[torch.Tensor, int]:
        sp_world_size = get_sequence_parallel_world_size()
        sp_rank = get_sp_parallel_rank()
        local_len = x_t.shape[1] // sp_world_size
        start = sp_rank * local_len
        end = start + local_len
        return x_t[:, start:end].contiguous(), start

    def sample_actions(
        self,
        observation: VLAObservationBatch,
        prefix_context: PrefixContext,
        *,
        noise: torch.Tensor | None,
        num_steps: int,
        use_cuda_graph: bool = True,

View on GitHub (pinned to 0132848349)

Solutions

1. Verify the distributed launch: world size, TP/SP settings, and that every rank enters sample_actions with the same action_sp_enabled flag
2. Ensure the action root rank actually holds/creates x_t before broadcast (check rank ordering logic)
3. Check NCCL logs for collective errors and set NCCL_DEBUG=INFO to diagnose
4. Reproduce single-rank to confirm the policy itself is healthy, then fix the distributed config

from sglang.multimodal_gen.runtime.models.vlas.pi05_policy import get_vla_split_group
split = get_vla_split_group()
assert split is not None, "VLA split group not initialized"
assert 0 <= split.action_root < dist.get_world_size(split.group), "action_root out of range"

try:
    actions = policy.sample_actions(prefix, suffix, x_t=x_t)
except RuntimeError as e:
    if "broadcast returned None" in str(e):
        # collective misconfiguration: fail fast with context, do not retry blindly
        raise RuntimeError(f"action broadcast failed on rank {dist.get_rank()}; "
                           "check SP/TP config and NCCL logs") from e
    raise

When it happens

Trigger: Running Pi05 with action sequence-parallelism where broadcast_tensor_from_rank returns None: mismatched world sizes, wrong src rank (action_root not part of the split group), CUDA/NCCL collective failure, or ranks taking divergent code paths so root never sent.

Common situations: Misconfigured TP/SP launch flags (e.g. --tp-size vs sequence-parallel size mismatch); one rank crashing or hitting a different branch; NCCL timeouts being swallowed; version skew between ranks or sglang nodes.

Related errors

• Pi05 action state is missing on single-rank run
• Pi05 action fallback must run on the action root
• UlyssesAttention's all-to-all spans the combined sequence pa
• --enable-tp-lm-head-all-to-all requires an available PyNCCL
• NCCL only supports CUDA, ROCm and MUSA backends.

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/fb9ad3d37728f446. Report an issue: GitHub.