sgl-project/sglang · error · RuntimeError

requires #senders == #recvs but got #senders={len(received_l

Error message

requires #senders == #recvs but got #senders={len(received_list)} vs #recvs={recv_world_size}

What it means

The Grafter's default transform picks, on each receiver, the tensor sent by the receiver's own rank; this only works in symmetric all-to-all collectives where the number of received tensors equals dist.get_world_size() of the receiving group. If fewer/more tensors were received, the default cannot index safely and raises.

Source

Thrown at python/sglang/srt/debug_utils/dumper.py:1064

            received_list=received_list,
            received_extras_list=received_extras_list,
            target=target,
        )
        path = self._config.grafter_transform_path
        fn = self._default_transform if path is None else _load_function(path)
        return fn(graft_input)

    @staticmethod
    def _default_transform(graft_input: GraftTransformInput) -> torch.Tensor:
        """Identity-by-rank fallback. Requires #senders == #recvs and
        shape(received_list[my_recv_rank]) == shape(target). Otherwise raises
        and asks the user for a transform."""
        received_list = graft_input.received_list
        target = graft_input.target
        my_recv_rank = dist.get_rank()
        recv_world_size = dist.get_world_size()
        if len(received_list) != recv_world_size:
            raise RuntimeError(
                _Grafter._default_transform_error(
                    f"requires #senders == #recvs but got "
                    f"#senders={len(received_list)} vs #recvs={recv_world_size}"
                )
            )
        candidate = received_list[my_recv_rank]
        if candidate.shape != target.shape:
            raise RuntimeError(
                _Grafter._default_transform_error(
                    f"requires matching shapes but "
                    f"received_list[{my_recv_rank}].shape={tuple(candidate.shape)} "
                    f"!= target.shape={tuple(target.shape)}"
                )
            )
        return candidate

    @staticmethod
    def _default_transform_error(detail: str) -> str:

View on GitHub (pinned to 0132848349)

Solutions

1. Supply an explicit transform that maps received_list + target to the desired tensor instead of relying on the default
2. Restrict grafting to collectives with equal sender/receiver counts
3. Verify dist.get_world_size() is queried on the correct process group for the intercepted op

Example fix

# before
cfg.grafter_transform = None  # default transform assumed
# after
cfg.grafter_transform = lambda graft_input: graft_input.target  # custom, no rank-index assumption

assert len(received_tensors) == dist.get_world_size(), "default transform needs symmetric all-to-all"

try:
    out = grafter.apply(...)
except RuntimeError as e:
    if "#senders == #recvs" in str(e):
        out = custom_transform(...)  # fallback mapping

When it happens

Trigger: An all-gather/all-to-all style op where the sender group size differs from the receiver group size (asymmetric EP/TP groups, custom dist op with a subset of senders), routed through the grafter with no user transform.

Common situations: Expert-parallel or asymmetric TP setups where senders and receivers are different process groups; using the grafter's default transform where a custom mapping is required.

Related errors

• Group {group_name} is destroyed.
• [Grafter] tags={tags} matched BOTH grafter_b2t_filter and gr
• requires matching shapes but received_list[{my_recv_rank}].s
• world_size must be positive and divide global_heads
• world_size ({world_size}) is less than tensor_parallel_degre

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/9567913709214c39. Report an issue: GitHub.