vllm-project/vllm · critical · RuntimeError

NCCL error: {error_str}

Error message

NCCL error: {error_str}

What it means

NCCLLibrary.NCCL_CHECK wraps every ctypes call into libnccl.so. Any ncclResult_t other than ncclSuccess is converted to a RuntimeError whose text comes from ncclGetErrorString (e.g. 'invalid argument', 'unhandled CUDA error', 'remote process exited'). The Python wrapper is only the messenger; the real failure is in the NCCL communicator or the network underneath it.

Source

Thrown at vllm/distributed/device_communicators/pynccl_wrapper.py:417

                            )
                        if current_platform.is_rocm():
                            # Having an exception here on ROCm platform is
                            # not allowed during graph capturing
                            continue
                    elif func.name == "ncclCommQueryProperties":
                        # Optional on NCCL versions older than 2.29.
                        continue
                    raise
            NCCLLibrary.path_to_dict_mapping[so_file] = _funcs
        self._funcs = NCCLLibrary.path_to_dict_mapping[so_file]

    def ncclGetErrorString(self, result: ncclResult_t) -> str:
        return self._funcs["ncclGetErrorString"](result).decode("utf-8")

    def NCCL_CHECK(self, result: ncclResult_t) -> None:
        if result != 0:
            error_str = self.ncclGetErrorString(result)
            raise RuntimeError(f"NCCL error: {error_str}")

    def ncclGetRawVersion(self) -> int:
        version = ctypes.c_int()
        self.NCCL_CHECK(self._funcs["ncclGetVersion"](ctypes.byref(version)))
        # something like 21903
        return version.value

    def ncclGetVersion(self) -> str:
        version_str = str(self.ncclGetRawVersion())
        # something like 21903 --> "2.19.3"
        major = version_str[0].lstrip("0")
        minor = version_str[1:3].lstrip("0")
        patch = version_str[3:].lstrip("0")
        return f"{major}.{minor}.{patch}"

    def ncclGetUniqueId(self) -> ncclUniqueId:
        unique_id = ncclUniqueId()
        self.NCCL_CHECK(self._funcs["ncclGetUniqueId"](ctypes.byref(unique_id)))

View on GitHub (pinned to c794754062)

Solutions

1. Re-run with NCCL_DEBUG=INFO (or TRACE) and read the first upstream error line above the Python traceback
2. Verify every rank passes the same world_size, rank and unique_id and issues identical collective shapes/dtypes
3. Check container network: set NCCL_SOCKET_IFNAME correctly, or increase shm (--ipc=host / --shm-size), and confirm IB devices are visible
4. Align the NCCL version across all nodes/containers (python -c 'import torch; torch.cuda.nccl.version()') and update CUDA drivers if ncclUnhandledCudaError appears

Example fix

# before
python -m vllm.entrypoints.openai.api_server ...  # opaque 'NCCL error: unhandled system error'

# after
export NCCL_DEBUG=INFO
python -m vllm.entrypoints.openai.api_server ...  # traceback now paired with root-cause NCCL log

import subprocess
free = int(subprocess.check_output(['df', '-B1', '/dev/shm']).splitlines()[-1].split()[3])
assert free > required_bytes, f"/dev/shm too small for NCCL shared buffers"

try:
    dist.all_reduce(x, group=g)
except RuntimeError as e:
    if "NCCL error" in str(e):
        log.exception("NCCL failure; enable NCCL_DEBUG=INFO, check peer liveness")
        raise  # NCCL communicators are usually unusable after failure; do not blind-retry

When it happens

Trigger: ncclCommInitRank/ncclCommInitRankConfig with mismatched world_size/unique_id across ranks; collectives issued on mismatched counts/dtypes/devices across ranks; a peer rank dying mid-collective (ncclRemoteError); IB/RDMA socket misconfiguration causing init failures (ncclSystemError).

Common situations: NCCL_SOCKET_IFNAME pointing at the wrong interface in containers; one worker OOM-crashing so survivors hang then fail; mixed NCCL versions across container images; firewall blocking the NCCL port range; mismatched tensor shapes across ranks in a custom collective.

Related errors

• DeepEPv2 communicator properties query failed; networking ca
• DeepEPv2 requires NCCL GIN (GPU-Initiated Networking). This
• Flashinfer allreduce is not supported for multi-node allredu
• Flashinfer allreduce quantization fusion is not supported fo
• Unsupported dtype {dtype}: should be one of int8, uint8, int

AI-assisted analysis of vllm-project/vllm@c794754062 (2026-08-14). Data as JSON: /api/errors/dc9ffae50b070960. Report an issue: GitHub.