{"record":{"id":"5ed532ce20f9d7cd","repo":"sgl-project/sglang","slug":"error-on-rank-0","errorCode":null,"errorMessage":"Error on rank 0","messagePattern":"Error on rank 0","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"critical","filePath":"python/sglang/multimodal_gen/runtime/pipelines_core/executors/parallel_executor.py","lineNumber":148,"sourceCode":"                                stage_index,\n                                batch,\n                                server_args,\n                                run_stage,\n                                use_nvtx,\n                            )\n                            obj_list = [True, batch]\n                        except Exception as e:\n                            obj_list = [False, e]\n\n                    # Send batch to other ranks\n                    broadcasted_list = broadcast_pyobj(\n                        obj_list, rank=rank, dist_group=group.cpu_group, src=0\n                    )\n                    success, broadcasted_batch = broadcasted_list\n\n                    if not success:\n                        if isinstance(broadcasted_batch, BaseException):\n                            raise RuntimeError(\"Error on rank 0\") from broadcasted_batch\n                        raise RuntimeError(f\"Error on rank 0: {broadcasted_batch}\")\n\n                    if rank != 0:\n                        batch = broadcasted_batch\n\n                    torch.distributed.barrier()\n        return batch\n\n    def execute(\n        self,\n        stages: List[PipelineStage],\n        batch: Req,\n        server_args: ServerArgs,\n    ) -> OutputBatch:\n        return self._execute_stages(\n            stages,\n            batch,\n            server_args,","sourceCodeStart":130,"sourceCodeEnd":166,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/multimodal_gen/runtime/pipelines_core/executors/parallel_executor.py#L130-L166","documentation":"In the parallel executor, non-zero ranks broadcast a (success, payload) tuple from rank 0 after executing stages. When rank 0 failed and the broadcast payload is an exception instance, this bare 'Error on rank 0' RuntimeError is raised with the rank-0 exception chained as __cause__. It exists to propagate rank-0 failures to all ranks so the group fails together rather than deadlocking.","triggerScenarios":"Running a pipeline under the parallel executor where stage execution on rank 0 raises any exception; that exception is pickled and broadcast, and every non-zero rank re-raises this wrapper via 'raise ... from broadcasted_batch'.","commonSituations":"OOM or CUDA error on rank 0, a shape/config mismatch that only manifests on rank 0's data, or a checkpoint loading failure on rank 0 in multi-GPU TI2V/diffusion serving. The root cause is in the chained exception ('from ...'), not this message.","solutions":["Inspect the chained exception (__cause__ / full traceback) to find the actual rank-0 error","Fix the underlying rank-0 failure (OOM, shape mismatch, missing weight, etc.)","Reproduce on a single process/rank to get a cleaner traceback of the same failure"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try:\n    batch = executor.execute(...)\nexcept RuntimeError as e:\n    if str(e).startswith(\"Error on rank 0\") and e.__cause__ is not None:\n        raise e.__cause__  # surface the real rank-0 exception\n    raise","preventionTips":["Log per-rank stage inputs/outputs to catch rank-0-only failures early","Validate shapes/paths identically on all ranks before entering the executor","Treat this wrapper as a pointer: always inspect __cause__ / the full traceback"],"tags":["distributed","multi-gpu","rank-failure","error-propagation"],"backgroundTag":"distributed-rank0-failure","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}