{"record":{"id":"805077b1f1b1c81d","repo":"vllm-project/vllm","slug":"nixl-eplb-metadata-key-mismatch-with-rank-peer","errorCode":null,"errorMessage":"NIXL EPLB metadata key mismatch with rank {peer}: local={sorted(local_keys)}, peer={sorted(peer_keys)}","messagePattern":"NIXL EPLB metadata key mismatch with rank (.+?): local=(.+?), peer=(.+?)","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"vllm/distributed/eplb/eplb_communicator.py","lineNumber":487,"sourceCode":"                )\n\n        # Per-rank map: (layer_idx, tensor_idx) -> (base_ptr, bytes_per_expert, dev_id).\n        # add_recv uses base_ptr + src_row * bytes_per_expert to compute\n        # the remote RDMA address for each expert.\n        gathered_meta: list[dict[tuple[int, int], tuple[int, int, int]] | None] = [\n            None\n        ] * self._world_size\n        torch.distributed.all_gather_object(\n            gathered_meta, local_meta, group=self._cpu_group\n        )\n\n        local_keys = set(local_meta.keys())\n        for peer in self._remote_agents:\n            peer_meta = gathered_meta[peer]\n            assert peer_meta is not None\n            peer_keys = set(peer_meta.keys())\n            if peer_keys != local_keys:\n                raise RuntimeError(\n                    f\"NIXL EPLB metadata key mismatch with rank {peer}: \"\n                    f\"local={sorted(local_keys)}, peer={sorted(peer_keys)}\"\n                )\n            for key in local_keys:\n                _, local_stride, _ = local_meta[key]\n                _, peer_stride, _ = peer_meta[key]\n                if local_stride != peer_stride:\n                    raise RuntimeError(\n                        f\"NIXL EPLB nbytes_per_expert mismatch for {key} \"\n                        f\"with rank {peer}: \"\n                        f\"local={local_stride}, peer={peer_stride}\"\n                    )\n            self._remote_send_meta[peer] = peer_meta\n\n    def _wait_for_all_transfers(self, handles: list[int]) -> None:\n        pending = set(handles)\n        while pending:\n            completed: list[int] = []","sourceCodeStart":469,"sourceCodeEnd":505,"githubUrl":"https://github.com/vllm-project/vllm/blob/c794754062d49a8fdb63ab3c5215b488b865030c/vllm/distributed/eplb/eplb_communicator.py#L469-L505","documentation":"Error \"NIXL EPLB metadata key mismatch with rank {peer}: local={sorted(local_keys)}, peer={sorted(peer_keys)}\" thrown in vllm-project/vllm.","triggerScenarios":"Raised at vllm/distributed/eplb/eplb_communicator.py:487 when validation fails: NIXL EPLB metadata key mismatch with peer rank. Typically triggered by an incompatible or incomplete vLLM configuration, an unsupported platform/backend combination, or a runtime resource/dependency that is missing.","commonSituations":"Commonly encountered at vllm/distributed/eplb/eplb_communicator.py:487 during vLLM startup/config validation or runtime setup when: (1) conflicting CLI flags or config fields are combined, (2) the current platform (CUDA/ROCm/CPU/XPU) or installed optional packages do not support the requested feature, or (3) a required value is absent or out of range. Resolve by correcting the configuration as described in the message, or by selecting a supported alternative.","solutions":["Ensure all ranks use the same EPLB configuration and register the same metadata keys.","Restart the deployment so local and peer keys match."],"exampleFix":null,"handlingStrategy":null,"validationCode":null,"typeGuard":null,"tryCatchPattern":null,"preventionTips":[],"tags":[],"backgroundTag":null,"analyzedSha":"c794754062d49a8fdb63ab3c5215b488b865030c","analyzedAt":"2026-08-14T21:17:39.825Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}