sgl-project/sglang · critical · RuntimeError

Error on rank 0: {broadcasted_batch}

Error message

Error on rank 0: {broadcasted_batch}

What it means

The non-exception variant of the rank-0 failure path in the parallel executor: rank 0 broadcast success=False with a non-exception payload (typically an error message string), so the wrapper includes that payload in the message. All ranks raise this so the distributed group fails consistently. The real problem is whatever rank 0 reported in broadcasted_batch.

Source

Thrown at python/sglang/multimodal_gen/runtime/pipelines_core/executors/parallel_executor.py:149

                                batch,
                                server_args,
                                run_stage,
                                use_nvtx,
                            )
                            obj_list = [True, batch]
                        except Exception as e:
                            obj_list = [False, e]

                    # Send batch to other ranks
                    broadcasted_list = broadcast_pyobj(
                        obj_list, rank=rank, dist_group=group.cpu_group, src=0
                    )
                    success, broadcasted_batch = broadcasted_list

                    if not success:
                        if isinstance(broadcasted_batch, BaseException):
                            raise RuntimeError("Error on rank 0") from broadcasted_batch
                        raise RuntimeError(f"Error on rank 0: {broadcasted_batch}")

                    if rank != 0:
                        batch = broadcasted_batch

                    torch.distributed.barrier()
        return batch

    def execute(
        self,
        stages: List[PipelineStage],
        batch: Req,
        server_args: ServerArgs,
    ) -> OutputBatch:
        return self._execute_stages(
            stages,
            batch,
            server_args,
            lambda stage, current: stage(current, server_args),

View on GitHub (pinned to 0132848349)

Solutions

1. Read the broadcasted_batch content in the message — it is rank 0's error description
2. Fix the underlying rank-0 condition described by that payload
3. Add per-rank logging around the executed stages to capture full rank-0 context

try:
    batch = executor.execute(...)
except RuntimeError as e:
    if str(e).startswith("Error on rank 0:"):
        logger.error("rank 0 reported: %s", str(e))
        # inspect payload in message; escalate/diagnose rank 0
    raise

When it happens

Trigger: Multi-rank pipeline execution where rank 0's stage returns/raises something serialized as a non-exception (e.g. a string error description); non-zero ranks then raise RuntimeError('Error on rank 0: <message>').

Common situations: Rank 0 hit an error that was caught and converted to a message string before broadcast; custom stages that return error strings; debugging multi-GPU diffusion pipelines where only rank 0 sees the failure (data-dependent or rank-0-only work).

Related errors

• Error on rank 0
• All inputs must be on the same device.
• All inputs must be on the same device.
• ReplaySSM inputs must be on the same device.
• {name} must be on device {device}, got {t.device}

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/0e1e64cc55dd236e. Report an issue: GitHub.