{"record":{"id":"a1bc6f7aa6a2c92b","repo":"sgl-project/sglang","slug":"requires-matching-shapes-but-received-list-my-rec","errorCode":null,"errorMessage":"requires matching shapes but received_list[{my_recv_rank}].shape={tuple(candidate.shape)} != target.shape={tuple(target.shape)}","messagePattern":"requires matching shapes but received_list\\[(.+?)\\]\\.shape=(.+?) != target\\.shape=(.+?)","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/debug_utils/dumper.py","lineNumber":1072,"sourceCode":"    @staticmethod\n    def _default_transform(graft_input: GraftTransformInput) -> torch.Tensor:\n        \"\"\"Identity-by-rank fallback. Requires #senders == #recvs and\n        shape(received_list[my_recv_rank]) == shape(target). Otherwise raises\n        and asks the user for a transform.\"\"\"\n        received_list = graft_input.received_list\n        target = graft_input.target\n        my_recv_rank = dist.get_rank()\n        recv_world_size = dist.get_world_size()\n        if len(received_list) != recv_world_size:\n            raise RuntimeError(\n                _Grafter._default_transform_error(\n                    f\"requires #senders == #recvs but got \"\n                    f\"#senders={len(received_list)} vs #recvs={recv_world_size}\"\n                )\n            )\n        candidate = received_list[my_recv_rank]\n        if candidate.shape != target.shape:\n            raise RuntimeError(\n                _Grafter._default_transform_error(\n                    f\"requires matching shapes but \"\n                    f\"received_list[{my_recv_rank}].shape={tuple(candidate.shape)} \"\n                    f\"!= target.shape={tuple(target.shape)}\"\n                )\n            )\n        return candidate\n\n    @staticmethod\n    def _default_transform_error(detail: str) -> str:\n        return (\n            f\"[Grafter] no grafter_transform_path set; default identity-by-rank \"\n            f\"{detail}. Provide a transform via \"\n            f\"DUMPER_GRAFTER_TRANSFORM_PATH=pkg.module.symbol defining \"\n            f\"`transform(graft_input: GraftTransformInput) -> Tensor`.\"\n        )\n\n","sourceCodeStart":1054,"sourceCodeEnd":1090,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/debug_utils/dumper.py#L1054-L1090","documentation":"The Grafter's default transform grafts received_list[my_rank] into the target, which presupposes sender i and receiver i produce identically-shaped tensors. If the selected candidate's shape differs from target.shape, the graft would silently broadcast-corrupt data, so it raises instead.","triggerScenarios":"A collective where per-rank tensors have different shapes (padding, ragged expert loads, variable batch chunking such as denoise/self-forcing chunks) and the default transform is used.","commonSituations":"Hybrid/padded sequences producing rank-local shapes; MoE expert imbalance; any non-uniform sharding where rank shapes are not identical.","solutions":["Provide a custom transform that reshapes/pads/slices the candidate to target's shape or picks a different source tensor","Ensure all ranks exchange identically-shaped tensors (pad to max shape) if you want the default transform","Log tuple(candidate.shape) vs tuple(target.shape) per rank first to confirm which side is off"],"exampleFix":"# before\ncfg.grafter_transform = None\n# after\ndef xform(g):\n    c = g.received_list[dist.get_rank()]\n    return c.reshape(g.target.shape) if c.numel() == g.target.numel() else g.target\ncfg.grafter_transform = xform","handlingStrategy":"fallback","validationCode":"cand = received_list[dist.get_rank()]\nif tuple(cand.shape) != tuple(target.shape):\n    # pad/reshape or choose custom transform before grafting\n    ...","typeGuard":null,"tryCatchPattern":"try:\n    out = grafter.apply(...)\nexcept RuntimeError as e:\n    if \"shape\" in str(e):\n        out = pad_or_reshape_transform(...)","preventionTips":["Pad rank-local tensors to a common shape before collectives","Log per-rank shapes during graft development"],"tags":["distributed","tensor-shape","grafter"],"backgroundTag":"tensor-shape-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}