sgl-project/sglang · error · RuntimeError

requires matching shapes but received_list[{my_recv_rank}].s

Error message

requires matching shapes but received_list[{my_recv_rank}].shape={tuple(candidate.shape)} != target.shape={tuple(target.shape)}

What it means

The Grafter's default transform grafts received_list[my_rank] into the target, which presupposes sender i and receiver i produce identically-shaped tensors. If the selected candidate's shape differs from target.shape, the graft would silently broadcast-corrupt data, so it raises instead.

Source

Thrown at python/sglang/srt/debug_utils/dumper.py:1072

    @staticmethod
    def _default_transform(graft_input: GraftTransformInput) -> torch.Tensor:
        """Identity-by-rank fallback. Requires #senders == #recvs and
        shape(received_list[my_recv_rank]) == shape(target). Otherwise raises
        and asks the user for a transform."""
        received_list = graft_input.received_list
        target = graft_input.target
        my_recv_rank = dist.get_rank()
        recv_world_size = dist.get_world_size()
        if len(received_list) != recv_world_size:
            raise RuntimeError(
                _Grafter._default_transform_error(
                    f"requires #senders == #recvs but got "
                    f"#senders={len(received_list)} vs #recvs={recv_world_size}"
                )
            )
        candidate = received_list[my_recv_rank]
        if candidate.shape != target.shape:
            raise RuntimeError(
                _Grafter._default_transform_error(
                    f"requires matching shapes but "
                    f"received_list[{my_recv_rank}].shape={tuple(candidate.shape)} "
                    f"!= target.shape={tuple(target.shape)}"
                )
            )
        return candidate

    @staticmethod
    def _default_transform_error(detail: str) -> str:
        return (
            f"[Grafter] no grafter_transform_path set; default identity-by-rank "
            f"{detail}. Provide a transform via "
            f"DUMPER_GRAFTER_TRANSFORM_PATH=pkg.module.symbol defining "
            f"`transform(graft_input: GraftTransformInput) -> Tensor`."
        )

View on GitHub (pinned to 0132848349)

Solutions

1. Provide a custom transform that reshapes/pads/slices the candidate to target's shape or picks a different source tensor
2. Ensure all ranks exchange identically-shaped tensors (pad to max shape) if you want the default transform
3. Log tuple(candidate.shape) vs tuple(target.shape) per rank first to confirm which side is off

Example fix

# before
cfg.grafter_transform = None
# after
def xform(g):
    c = g.received_list[dist.get_rank()]
    return c.reshape(g.target.shape) if c.numel() == g.target.numel() else g.target
cfg.grafter_transform = xform

cand = received_list[dist.get_rank()]
if tuple(cand.shape) != tuple(target.shape):
    # pad/reshape or choose custom transform before grafting
    ...

try:
    out = grafter.apply(...)
except RuntimeError as e:
    if "shape" in str(e):
        out = pad_or_reshape_transform(...)

When it happens

Trigger: A collective where per-rank tensors have different shapes (padding, ragged expert loads, variable batch chunking such as denoise/self-forcing chunks) and the default transform is used.

Common situations: Hybrid/padded sequences producing rank-local shapes; MoE expert imbalance; any non-uniform sharding where rank shapes are not identical.

Related errors

• [Grafter] tags={tags} matched BOTH grafter_b2t_filter and gr
• requires #senders == #recvs but got #senders={len(received_l
• No frames were recorded
• The batch size is expected to be 1 rather than {q.shape[0]}
• `dt_bias` must have {HV * K} elements (got {dt_bias.numel()}

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/a1bc6f7aa6a2c92b. Report an issue: GitHub.