vllm-project/vllm · critical · RuntimeError

DeepEPv2 communicator properties query failed; networking ca

Error message

DeepEPv2 communicator properties query failed; networking capability could not be determined.

What it means

Raised by _check_gin_support in the DeepEPv2 all2all path when query_nccl_gin_type(group) returns None after the group was explicitly initialized with an all_reduce probe. Returning None means the NCCL communicator properties could not be read at all (comm still null or the query API failed), so networking capability is indeterminate and vLLM refuses to continue.

Source

Thrown at vllm/distributed/device_communicators/all2all.py:1054

            use_fp8_dispatch=use_fp8_dispatch,
            allow_hybrid_mode=envs.VLLM_DEEPEP_V2_ALLOW_HYBRID_MODE,
            prefer_overlap_with_compute=envs.VLLM_DEEPEP_V2_PREFER_OVERLAP,
            allow_multiple_reduction=(envs.VLLM_DEEPEP_V2_ALLOW_MULTIPLE_REDUCTION),
            explicitly_destroy=True,
        )

    def _check_gin_support(self, group) -> None:
        from vllm.utils.nccl import query_nccl_gin_type

        # ProcessGroupNCCL creates communicators lazily. Initialize this exact
        # group before querying so a null comm pointer is not mistaken for
        # missing GIN support.
        probe = torch.zeros(1, device="cuda")
        torch.distributed.all_reduce(probe, group=group)

        gin_type = query_nccl_gin_type(group)
        if gin_type is None:
            raise RuntimeError(
                "DeepEPv2 communicator properties query failed; "
                "networking capability could not be determined."
            )
        if gin_type == 0:
            raise RuntimeError(
                "DeepEPv2 requires NCCL GIN (GPU-Initiated Networking). "
                "This usually means IBGDA-capable InfiniBand NICs or drivers "
                "are not available. See tools/ep_kernels/README.md for "
                "requirements."
            )

    def get_handle(self, kwargs):
        import deep_ep  # type: ignore[import-not-found]

        num_experts = kwargs.pop("num_experts", 256)
        buffer_kwargs = self._make_all2all_kwargs(**kwargs)
        if not self._gin_checked:
            self._check_gin_support(buffer_kwargs["group"])

View on GitHub (pinned to c794754062)

Solutions

Ensure the correct GPU process group (the one used for the EP collectives) is passed, not the CPU/world group.
Use a vLLM-recommended NCCL build/version (e.g. the bundled vllm-nccl or a NCCL release that exposes the GIN/IBGDA property query) and retry.
Reproduce the query standalone (query_nccl_gin_type on your group after an all_reduce) to see whether the property API exists in your NCCL; if not, upgrade NCCL.
If the stack genuinely lacks GIN support, DeepEPv2 cannot be used on this cluster — switch the all2all backend.

Example fix

# before: cpu_group passed to the DeepEPv2 manager -> query returns None
# after: pass the GPU (device) group used for EP collectives
manager = DeepEPv2Manager(gpu_group)

Defensive patterns

Strategy: try-catch

Validate before calling

from vllm.utils.nccl import query_nccl_gin_type
import torch

probe = torch.zeros(1, device="cuda")
torch.distributed.all_reduce(probe, group=gpu_group)
if query_nccl_gin_type(gpu_group) is None:
    raise SystemExit("NCCL comm query failed; check NCCL build/version and group type")

Type guard

def gin_type_queryable(group) -> bool:
    from vllm.utils.nccl import query_nccl_gin_type
    probe = torch.zeros(1, device="cuda")
    torch.distributed.all_reduce(probe, group=group)
    return query_nccl_gin_type(group) is not None

Try / catch

try:
    manager = DeepEPv2Manager(gpu_group)
except RuntimeError as e:
    if "properties query failed" in str(e):
        verify_gpu_group_and_nccl_build_then_abort()  # not retryable as-is
    raise

Prevention

Pass the device (GPU) process group, never the CPU/world group, to EP communicators.
Standardize on a vLLM-supported NCCL build (bundled vllm-nccl or documented minimum) cluster-wide.
Add a cluster admission test that queries GIN capability before scheduling DeepEP workloads.

When it happens

Trigger: Constructing the DeepEPv2 all2all manager with a ProcessGroupNCCL group whose communicator cannot be queried for its GIN type — e.g. the group was not actually used for collectives, an NCCL version lacking the property query, or a mismatched NCCL build.

Common situations: Custom NCCL builds (vLLM's bundled nccl vs system nccl) missing the config query symbols; the CPU/other-process group being passed instead of the GPU group; NCCL_TOO_OLD or forked communicators where the query returns nothing.

Related errors

AI-assisted analysis of vllm-project/vllm@c794754062 (2026-08-14). Data as JSON: /api/errors/b4ad97cc05cb3165. Report an issue: GitHub.