sgl-project/sglang · error · RuntimeError

Pi05 action fallback must run on the action root

Error message

Pi05 action fallback must run on the action root

What it means

When action sequence-parallelism is disabled (action_sp_enabled False), sample_actions falls back to running the full denoising loop only on the action-root rank, which must therefore have a locally-created x_t. If the code reaches the fallback branch with x_t still None, the rank has neither received a broadcast nor built its own initial noise, so execution cannot proceed. This typically means the caller is a non-root rank on the fallback path.

Source

Thrown at python/sglang/multimodal_gen/runtime/models/vlas/pi05_policy.py:1133

            use_cuda_graph = False

        split = get_vla_split_group()
        action_sp_enabled = self._can_use_action_sequence_parallel(
            prefix_context,
            self.config.action_horizon,
        )
        if split is None or split.rank == split.action_root:
            x_t = noise
            if x_t is None:
                x_t = self.sample_noise(observation.batch_size, generator=generator)
            else:
                x_t = x_t.to(device=self.device, dtype=torch.float32).clone()
        else:
            x_t = None
        if action_sp_enabled:
            x_t = self._broadcast_initial_action_state(x_t)
        elif x_t is None:
            raise RuntimeError("Pi05 action fallback must run on the action root")
        action_position_offset = 0
        if action_sp_enabled:
            x_t, action_position_offset = self._shard_action_sequence(x_t)

        dt = -1.0 / num_steps
        timesteps = torch.linspace(
            1.0,
            1.0 / num_steps,
            num_steps,
            dtype=torch.float32,
            device=self.device,
        )
        for timestep_value in timesteps:
            timestep = timestep_value.expand(observation.batch_size)
            velocity = self.denoise_step(
                prefix_context,
                x_t,
                timestep,

View on GitHub (pinned to 0132848349)

Solutions

1. Enable action sequence parallelism for multi-rank Pi05 runs so all ranks receive the broadcast x_t
2. Or route sample_actions only through the action-root rank (workers should skip or gather from root)
3. If calling the API directly on the root, ensure the noise-initialization branch conditions are met so x_t is created
4. Upgrade sglang in case the root/fallback routing logic was fixed

is_action_root = (get_sp_parallel_rank() == 0)  # or your action-root predicate
if not action_sp_enabled and not is_action_root:
    skip_sample_actions = True  # route through root instead of calling directly

try:
    actions = policy.sample_actions(prefix, suffix, x_t=x_t)
except RuntimeError as e:
    if "action fallback must run on the action root" in str(e):
        # only the root should compute; others must gather results instead
        if not is_action_root:
            actions = gather_from_action_root()
        else:
            raise
    else:
        raise

When it happens

Trigger: Calling sample_actions (directly, or via run_grouped_requests/forward) on a rank where action SP is disabled and the branch that constructs x_t (the noise initialization above) did not execute — e.g. a non-action-root worker rank or an inconsistent x_t-passing convention.

Common situations: Running multi-rank Pi05 without action sequence-parallel enabled so worker ranks also enter sample_actions; calling forward on a DP worker that isn't the action root; inconsistent guard conditions between the x_t-creation branch and the action_sp_enabled flag.

Related errors

• Pi05 action state is missing on single-rank run
• Pi05 action state broadcast returned None
• UlyssesAttention's all-to-all spans the combined sequence pa
• world_size must be positive and divide global_heads
• Group {group_name} is destroyed.

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/b46ff51071b23d3e. Report an issue: GitHub.