{"record":{"id":"0e1e64cc55dd236e","repo":"sgl-project/sglang","slug":"error-on-rank-0-broadcasted-batch","errorCode":null,"errorMessage":"Error on rank 0: {broadcasted_batch}","messagePattern":"Error on rank 0: (.+?)","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"critical","filePath":"python/sglang/multimodal_gen/runtime/pipelines_core/executors/parallel_executor.py","lineNumber":149,"sourceCode":"                                batch,\n                                server_args,\n                                run_stage,\n                                use_nvtx,\n                            )\n                            obj_list = [True, batch]\n                        except Exception as e:\n                            obj_list = [False, e]\n\n                    # Send batch to other ranks\n                    broadcasted_list = broadcast_pyobj(\n                        obj_list, rank=rank, dist_group=group.cpu_group, src=0\n                    )\n                    success, broadcasted_batch = broadcasted_list\n\n                    if not success:\n                        if isinstance(broadcasted_batch, BaseException):\n                            raise RuntimeError(\"Error on rank 0\") from broadcasted_batch\n                        raise RuntimeError(f\"Error on rank 0: {broadcasted_batch}\")\n\n                    if rank != 0:\n                        batch = broadcasted_batch\n\n                    torch.distributed.barrier()\n        return batch\n\n    def execute(\n        self,\n        stages: List[PipelineStage],\n        batch: Req,\n        server_args: ServerArgs,\n    ) -> OutputBatch:\n        return self._execute_stages(\n            stages,\n            batch,\n            server_args,\n            lambda stage, current: stage(current, server_args),","sourceCodeStart":131,"sourceCodeEnd":167,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/multimodal_gen/runtime/pipelines_core/executors/parallel_executor.py#L131-L167","documentation":"The non-exception variant of the rank-0 failure path in the parallel executor: rank 0 broadcast success=False with a non-exception payload (typically an error message string), so the wrapper includes that payload in the message. All ranks raise this so the distributed group fails consistently. The real problem is whatever rank 0 reported in broadcasted_batch.","triggerScenarios":"Multi-rank pipeline execution where rank 0's stage returns/raises something serialized as a non-exception (e.g. a string error description); non-zero ranks then raise RuntimeError('Error on rank 0: <message>').","commonSituations":"Rank 0 hit an error that was caught and converted to a message string before broadcast; custom stages that return error strings; debugging multi-GPU diffusion pipelines where only rank 0 sees the failure (data-dependent or rank-0-only work).","solutions":["Read the broadcasted_batch content in the message — it is rank 0's error description","Fix the underlying rank-0 condition described by that payload","Add per-rank logging around the executed stages to capture full rank-0 context"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try:\n    batch = executor.execute(...)\nexcept RuntimeError as e:\n    if str(e).startswith(\"Error on rank 0:\"):\n        logger.error(\"rank 0 reported: %s\", str(e))\n        # inspect payload in message; escalate/diagnose rank 0\n    raise","preventionTips":["Never convert exceptions to plain strings in custom stages; re-raise so the executor chains them","Run a single-rank smoke test before multi-rank execution"],"tags":["distributed","multi-gpu","rank-failure","error-propagation"],"backgroundTag":"distributed-rank0-failure","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}