vllm-project/vllm · error · RuntimeError

NIXL EPLB metadata key mismatch with rank {peer}: local={sor

Error message

NIXL EPLB metadata key mismatch with rank {peer}: local={sorted(local_keys)}, peer={sorted(peer_keys)}

What it means

Error "NIXL EPLB metadata key mismatch with rank {peer}: local={sorted(local_keys)}, peer={sorted(peer_keys)}" thrown in vllm-project/vllm.

Source

Thrown at vllm/distributed/eplb/eplb_communicator.py:487

                )

        # Per-rank map: (layer_idx, tensor_idx) -> (base_ptr, bytes_per_expert, dev_id).
        # add_recv uses base_ptr + src_row * bytes_per_expert to compute
        # the remote RDMA address for each expert.
        gathered_meta: list[dict[tuple[int, int], tuple[int, int, int]] | None] = [
            None
        ] * self._world_size
        torch.distributed.all_gather_object(
            gathered_meta, local_meta, group=self._cpu_group
        )

        local_keys = set(local_meta.keys())
        for peer in self._remote_agents:
            peer_meta = gathered_meta[peer]
            assert peer_meta is not None
            peer_keys = set(peer_meta.keys())
            if peer_keys != local_keys:
                raise RuntimeError(
                    f"NIXL EPLB metadata key mismatch with rank {peer}: "
                    f"local={sorted(local_keys)}, peer={sorted(peer_keys)}"
                )
            for key in local_keys:
                _, local_stride, _ = local_meta[key]
                _, peer_stride, _ = peer_meta[key]
                if local_stride != peer_stride:
                    raise RuntimeError(
                        f"NIXL EPLB nbytes_per_expert mismatch for {key} "
                        f"with rank {peer}: "
                        f"local={local_stride}, peer={peer_stride}"
                    )
            self._remote_send_meta[peer] = peer_meta

    def _wait_for_all_transfers(self, handles: list[int]) -> None:
        pending = set(handles)
        while pending:
            completed: list[int] = []

View on GitHub (pinned to c794754062)

Solutions

1. Ensure all ranks use the same EPLB configuration and register the same metadata keys.
2. Restart the deployment so local and peer keys match.

When it happens

Trigger: Raised at vllm/distributed/eplb/eplb_communicator.py:487 when validation fails: NIXL EPLB metadata key mismatch with peer rank. Typically triggered by an incompatible or incomplete vLLM configuration, an unsupported platform/backend combination, or a runtime resource/dependency that is missing.

Common situations: Commonly encountered at vllm/distributed/eplb/eplb_communicator.py:487 during vLLM startup/config validation or runtime setup when: (1) conflicting CLI flags or config fields are combined, (2) the current platform (CUDA/ROCm/CPU/XPU) or installed optional packages do not support the requested feature, or (3) a required value is absent or out of range. Resolve by correcting the configuration as described in the message, or by selecting a supported alternative.

AI-assisted analysis of vllm-project/vllm@c794754062 (2026-08-14). Data as JSON: /api/errors/805077b1f1b1c81d. Report an issue: GitHub.