{"record":{"id":"9567913709214c39","repo":"sgl-project/sglang","slug":"requires-senders-recvs-but-got-senders-len","errorCode":null,"errorMessage":"requires #senders == #recvs but got #senders={len(received_list)} vs #recvs={recv_world_size}","messagePattern":"requires #senders == #recvs but got #senders=(.+?) vs #recvs=(.+?)","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/debug_utils/dumper.py","lineNumber":1064,"sourceCode":"            received_list=received_list,\n            received_extras_list=received_extras_list,\n            target=target,\n        )\n        path = self._config.grafter_transform_path\n        fn = self._default_transform if path is None else _load_function(path)\n        return fn(graft_input)\n\n    @staticmethod\n    def _default_transform(graft_input: GraftTransformInput) -> torch.Tensor:\n        \"\"\"Identity-by-rank fallback. Requires #senders == #recvs and\n        shape(received_list[my_recv_rank]) == shape(target). Otherwise raises\n        and asks the user for a transform.\"\"\"\n        received_list = graft_input.received_list\n        target = graft_input.target\n        my_recv_rank = dist.get_rank()\n        recv_world_size = dist.get_world_size()\n        if len(received_list) != recv_world_size:\n            raise RuntimeError(\n                _Grafter._default_transform_error(\n                    f\"requires #senders == #recvs but got \"\n                    f\"#senders={len(received_list)} vs #recvs={recv_world_size}\"\n                )\n            )\n        candidate = received_list[my_recv_rank]\n        if candidate.shape != target.shape:\n            raise RuntimeError(\n                _Grafter._default_transform_error(\n                    f\"requires matching shapes but \"\n                    f\"received_list[{my_recv_rank}].shape={tuple(candidate.shape)} \"\n                    f\"!= target.shape={tuple(target.shape)}\"\n                )\n            )\n        return candidate\n\n    @staticmethod\n    def _default_transform_error(detail: str) -> str:","sourceCodeStart":1046,"sourceCodeEnd":1082,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/debug_utils/dumper.py#L1046-L1082","documentation":"The Grafter's default transform picks, on each receiver, the tensor sent by the receiver's own rank; this only works in symmetric all-to-all collectives where the number of received tensors equals dist.get_world_size() of the receiving group. If fewer/more tensors were received, the default cannot index safely and raises.","triggerScenarios":"An all-gather/all-to-all style op where the sender group size differs from the receiver group size (asymmetric EP/TP groups, custom dist op with a subset of senders), routed through the grafter with no user transform.","commonSituations":"Expert-parallel or asymmetric TP setups where senders and receivers are different process groups; using the grafter's default transform where a custom mapping is required.","solutions":["Supply an explicit transform that maps received_list + target to the desired tensor instead of relying on the default","Restrict grafting to collectives with equal sender/receiver counts","Verify dist.get_world_size() is queried on the correct process group for the intercepted op"],"exampleFix":"# before\ncfg.grafter_transform = None  # default transform assumed\n# after\ncfg.grafter_transform = lambda graft_input: graft_input.target  # custom, no rank-index assumption","handlingStrategy":"fallback","validationCode":"assert len(received_tensors) == dist.get_world_size(), \"default transform needs symmetric all-to-all\"","typeGuard":null,"tryCatchPattern":"try:\n    out = grafter.apply(...)\nexcept RuntimeError as e:\n    if \"#senders == #recvs\" in str(e):\n        out = custom_transform(...)  # fallback mapping","preventionTips":["Always supply an explicit transform for asymmetric collectives","Confirm which process group get_world_size() refers to before relying on the default"],"tags":["distributed","collective","grafter"],"backgroundTag":"collective-group-size-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}