{"record":{"id":"b46ff51071b23d3e","repo":"sgl-project/sglang","slug":"pi05-action-fallback-must-run-on-the-action-root","errorCode":null,"errorMessage":"Pi05 action fallback must run on the action root","messagePattern":"Pi05 action fallback must run on the action root","errorType":"error_code","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"python/sglang/multimodal_gen/runtime/models/vlas/pi05_policy.py","lineNumber":1133,"sourceCode":"            use_cuda_graph = False\n\n        split = get_vla_split_group()\n        action_sp_enabled = self._can_use_action_sequence_parallel(\n            prefix_context,\n            self.config.action_horizon,\n        )\n        if split is None or split.rank == split.action_root:\n            x_t = noise\n            if x_t is None:\n                x_t = self.sample_noise(observation.batch_size, generator=generator)\n            else:\n                x_t = x_t.to(device=self.device, dtype=torch.float32).clone()\n        else:\n            x_t = None\n        if action_sp_enabled:\n            x_t = self._broadcast_initial_action_state(x_t)\n        elif x_t is None:\n            raise RuntimeError(\"Pi05 action fallback must run on the action root\")\n        action_position_offset = 0\n        if action_sp_enabled:\n            x_t, action_position_offset = self._shard_action_sequence(x_t)\n\n        dt = -1.0 / num_steps\n        timesteps = torch.linspace(\n            1.0,\n            1.0 / num_steps,\n            num_steps,\n            dtype=torch.float32,\n            device=self.device,\n        )\n        for timestep_value in timesteps:\n            timestep = timestep_value.expand(observation.batch_size)\n            velocity = self.denoise_step(\n                prefix_context,\n                x_t,\n                timestep,","sourceCodeStart":1115,"sourceCodeEnd":1151,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/multimodal_gen/runtime/models/vlas/pi05_policy.py#L1115-L1151","documentation":"When action sequence-parallelism is disabled (action_sp_enabled False), sample_actions falls back to running the full denoising loop only on the action-root rank, which must therefore have a locally-created x_t. If the code reaches the fallback branch with x_t still None, the rank has neither received a broadcast nor built its own initial noise, so execution cannot proceed. This typically means the caller is a non-root rank on the fallback path.","triggerScenarios":"Calling sample_actions (directly, or via run_grouped_requests/forward) on a rank where action SP is disabled and the branch that constructs x_t (the noise initialization above) did not execute — e.g. a non-action-root worker rank or an inconsistent x_t-passing convention.","commonSituations":"Running multi-rank Pi05 without action sequence-parallel enabled so worker ranks also enter sample_actions; calling forward on a DP worker that isn't the action root; inconsistent guard conditions between the x_t-creation branch and the action_sp_enabled flag.","solutions":["Enable action sequence parallelism for multi-rank Pi05 runs so all ranks receive the broadcast x_t","Or route sample_actions only through the action-root rank (workers should skip or gather from root)","If calling the API directly on the root, ensure the noise-initialization branch conditions are met so x_t is created","Upgrade sglang in case the root/fallback routing logic was fixed"],"exampleFix":null,"handlingStrategy":"validation","validationCode":"is_action_root = (get_sp_parallel_rank() == 0)  # or your action-root predicate\nif not action_sp_enabled and not is_action_root:\n    skip_sample_actions = True  # route through root instead of calling directly","typeGuard":null,"tryCatchPattern":"try:\n    actions = policy.sample_actions(prefix, suffix, x_t=x_t)\nexcept RuntimeError as e:\n    if \"action fallback must run on the action root\" in str(e):\n        # only the root should compute; others must gather results instead\n        if not is_action_root:\n            actions = gather_from_action_root()\n        else:\n            raise\n    else:\n        raise","preventionTips":["Enable action sequence parallelism whenever running Pi05 across multiple ranks","Gate sample_actions calls behind an is_action_root check in your wrapper code","Keep single-rank and multi-rank code paths behind the same integration test"],"tags":["pi05","distributed","action-root","sequence-parallel"],"backgroundTag":"distributed-rank-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}