sgl-project/sglang · critical · RuntimeError

Error on rank 0

Error message

Error on rank 0

What it means

In the parallel executor, non-zero ranks broadcast a (success, payload) tuple from rank 0 after executing stages. When rank 0 failed and the broadcast payload is an exception instance, this bare 'Error on rank 0' RuntimeError is raised with the rank-0 exception chained as __cause__. It exists to propagate rank-0 failures to all ranks so the group fails together rather than deadlocking.

Source

Thrown at python/sglang/multimodal_gen/runtime/pipelines_core/executors/parallel_executor.py:148

                                stage_index,
                                batch,
                                server_args,
                                run_stage,
                                use_nvtx,
                            )
                            obj_list = [True, batch]
                        except Exception as e:
                            obj_list = [False, e]

                    # Send batch to other ranks
                    broadcasted_list = broadcast_pyobj(
                        obj_list, rank=rank, dist_group=group.cpu_group, src=0
                    )
                    success, broadcasted_batch = broadcasted_list

                    if not success:
                        if isinstance(broadcasted_batch, BaseException):
                            raise RuntimeError("Error on rank 0") from broadcasted_batch
                        raise RuntimeError(f"Error on rank 0: {broadcasted_batch}")

                    if rank != 0:
                        batch = broadcasted_batch

                    torch.distributed.barrier()
        return batch

    def execute(
        self,
        stages: List[PipelineStage],
        batch: Req,
        server_args: ServerArgs,
    ) -> OutputBatch:
        return self._execute_stages(
            stages,
            batch,
            server_args,

View on GitHub (pinned to 0132848349)

Solutions

1. Inspect the chained exception (__cause__ / full traceback) to find the actual rank-0 error
2. Fix the underlying rank-0 failure (OOM, shape mismatch, missing weight, etc.)
3. Reproduce on a single process/rank to get a cleaner traceback of the same failure

try:
    batch = executor.execute(...)
except RuntimeError as e:
    if str(e).startswith("Error on rank 0") and e.__cause__ is not None:
        raise e.__cause__  # surface the real rank-0 exception
    raise

When it happens

Trigger: Running a pipeline under the parallel executor where stage execution on rank 0 raises any exception; that exception is pickled and broadcast, and every non-zero rank re-raises this wrapper via 'raise ... from broadcasted_batch'.

Common situations: OOM or CUDA error on rank 0, a shape/config mismatch that only manifests on rank 0's data, or a checkpoint loading failure on rank 0 in multi-GPU TI2V/diffusion serving. The root cause is in the chained exception ('from ...'), not this message.

Related errors

• Error on rank 0: {broadcasted_batch}
• All inputs must be on the same device.
• All inputs must be on the same device.
• ReplaySSM inputs must be on the same device.
• {name} must be on device {device}, got {t.device}

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/5ed532ce20f9d7cd. Report an issue: GitHub.