{"record":{"id":"b4ad97cc05cb3165","repo":"vllm-project/vllm","slug":"deepepv2-communicator-properties-query-failed-net","errorCode":null,"errorMessage":"DeepEPv2 communicator properties query failed; networking capability could not be determined.","messagePattern":"DeepEPv2 communicator properties query failed; networking capability could not be determined\\.","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"critical","filePath":"vllm/distributed/device_communicators/all2all.py","lineNumber":1054,"sourceCode":"            use_fp8_dispatch=use_fp8_dispatch,\n            allow_hybrid_mode=envs.VLLM_DEEPEP_V2_ALLOW_HYBRID_MODE,\n            prefer_overlap_with_compute=envs.VLLM_DEEPEP_V2_PREFER_OVERLAP,\n            allow_multiple_reduction=(envs.VLLM_DEEPEP_V2_ALLOW_MULTIPLE_REDUCTION),\n            explicitly_destroy=True,\n        )\n\n    def _check_gin_support(self, group) -> None:\n        from vllm.utils.nccl import query_nccl_gin_type\n\n        # ProcessGroupNCCL creates communicators lazily. Initialize this exact\n        # group before querying so a null comm pointer is not mistaken for\n        # missing GIN support.\n        probe = torch.zeros(1, device=\"cuda\")\n        torch.distributed.all_reduce(probe, group=group)\n\n        gin_type = query_nccl_gin_type(group)\n        if gin_type is None:\n            raise RuntimeError(\n                \"DeepEPv2 communicator properties query failed; \"\n                \"networking capability could not be determined.\"\n            )\n        if gin_type == 0:\n            raise RuntimeError(\n                \"DeepEPv2 requires NCCL GIN (GPU-Initiated Networking). \"\n                \"This usually means IBGDA-capable InfiniBand NICs or drivers \"\n                \"are not available. See tools/ep_kernels/README.md for \"\n                \"requirements.\"\n            )\n\n    def get_handle(self, kwargs):\n        import deep_ep  # type: ignore[import-not-found]\n\n        num_experts = kwargs.pop(\"num_experts\", 256)\n        buffer_kwargs = self._make_all2all_kwargs(**kwargs)\n        if not self._gin_checked:\n            self._check_gin_support(buffer_kwargs[\"group\"])","sourceCodeStart":1036,"sourceCodeEnd":1072,"githubUrl":"https://github.com/vllm-project/vllm/blob/c794754062d49a8fdb63ab3c5215b488b865030c/vllm/distributed/device_communicators/all2all.py#L1036-L1072","documentation":"Raised by _check_gin_support in the DeepEPv2 all2all path when query_nccl_gin_type(group) returns None after the group was explicitly initialized with an all_reduce probe. Returning None means the NCCL communicator properties could not be read at all (comm still null or the query API failed), so networking capability is indeterminate and vLLM refuses to continue.","triggerScenarios":"Constructing the DeepEPv2 all2all manager with a ProcessGroupNCCL group whose communicator cannot be queried for its GIN type — e.g. the group was not actually used for collectives, an NCCL version lacking the property query, or a mismatched NCCL build.","commonSituations":"Custom NCCL builds (vLLM's bundled nccl vs system nccl) missing the config query symbols; the CPU/other-process group being passed instead of the GPU group; NCCL_TOO_OLD or forked communicators where the query returns nothing.","solutions":["Ensure the correct GPU process group (the one used for the EP collectives) is passed, not the CPU/world group.","Use a vLLM-recommended NCCL build/version (e.g. the bundled vllm-nccl or a NCCL release that exposes the GIN/IBGDA property query) and retry.","Reproduce the query standalone (query_nccl_gin_type on your group after an all_reduce) to see whether the property API exists in your NCCL; if not, upgrade NCCL.","If the stack genuinely lacks GIN support, DeepEPv2 cannot be used on this cluster — switch the all2all backend."],"exampleFix":"# before: cpu_group passed to the DeepEPv2 manager -> query returns None\n# after: pass the GPU (device) group used for EP collectives\nmanager = DeepEPv2Manager(gpu_group)","handlingStrategy":"try-catch","validationCode":"from vllm.utils.nccl import query_nccl_gin_type\nimport torch\n\nprobe = torch.zeros(1, device=\"cuda\")\ntorch.distributed.all_reduce(probe, group=gpu_group)\nif query_nccl_gin_type(gpu_group) is None:\n    raise SystemExit(\"NCCL comm query failed; check NCCL build/version and group type\")","typeGuard":"def gin_type_queryable(group) -> bool:\n    from vllm.utils.nccl import query_nccl_gin_type\n    probe = torch.zeros(1, device=\"cuda\")\n    torch.distributed.all_reduce(probe, group=group)\n    return query_nccl_gin_type(group) is not None","tryCatchPattern":"try:\n    manager = DeepEPv2Manager(gpu_group)\nexcept RuntimeError as e:\n    if \"properties query failed\" in str(e):\n        verify_gpu_group_and_nccl_build_then_abort()  # not retryable as-is\n    raise","preventionTips":["Pass the device (GPU) process group, never the CPU/world group, to EP communicators.","Standardize on a vLLM-supported NCCL build (bundled vllm-nccl or documented minimum) cluster-wide.","Add a cluster admission test that queries GIN capability before scheduling DeepEP workloads."],"tags":["vllm","deepep","nccl","networking","distributed"],"backgroundTag":null,"analyzedSha":"c794754062d49a8fdb63ab3c5215b488b865030c","analyzedAt":"2026-08-14T21:17:39.825Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}