{"record":{"id":"dc9ffae50b070960","repo":"vllm-project/vllm","slug":"nccl-error-error-str","errorCode":null,"errorMessage":"NCCL error: {error_str}","messagePattern":"NCCL error: (.+?)","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"critical","filePath":"vllm/distributed/device_communicators/pynccl_wrapper.py","lineNumber":417,"sourceCode":"                            )\n                        if current_platform.is_rocm():\n                            # Having an exception here on ROCm platform is\n                            # not allowed during graph capturing\n                            continue\n                    elif func.name == \"ncclCommQueryProperties\":\n                        # Optional on NCCL versions older than 2.29.\n                        continue\n                    raise\n            NCCLLibrary.path_to_dict_mapping[so_file] = _funcs\n        self._funcs = NCCLLibrary.path_to_dict_mapping[so_file]\n\n    def ncclGetErrorString(self, result: ncclResult_t) -> str:\n        return self._funcs[\"ncclGetErrorString\"](result).decode(\"utf-8\")\n\n    def NCCL_CHECK(self, result: ncclResult_t) -> None:\n        if result != 0:\n            error_str = self.ncclGetErrorString(result)\n            raise RuntimeError(f\"NCCL error: {error_str}\")\n\n    def ncclGetRawVersion(self) -> int:\n        version = ctypes.c_int()\n        self.NCCL_CHECK(self._funcs[\"ncclGetVersion\"](ctypes.byref(version)))\n        # something like 21903\n        return version.value\n\n    def ncclGetVersion(self) -> str:\n        version_str = str(self.ncclGetRawVersion())\n        # something like 21903 --> \"2.19.3\"\n        major = version_str[0].lstrip(\"0\")\n        minor = version_str[1:3].lstrip(\"0\")\n        patch = version_str[3:].lstrip(\"0\")\n        return f\"{major}.{minor}.{patch}\"\n\n    def ncclGetUniqueId(self) -> ncclUniqueId:\n        unique_id = ncclUniqueId()\n        self.NCCL_CHECK(self._funcs[\"ncclGetUniqueId\"](ctypes.byref(unique_id)))","sourceCodeStart":399,"sourceCodeEnd":435,"githubUrl":"https://github.com/vllm-project/vllm/blob/c794754062d49a8fdb63ab3c5215b488b865030c/vllm/distributed/device_communicators/pynccl_wrapper.py#L399-L435","documentation":"NCCLLibrary.NCCL_CHECK wraps every ctypes call into libnccl.so. Any ncclResult_t other than ncclSuccess is converted to a RuntimeError whose text comes from ncclGetErrorString (e.g. 'invalid argument', 'unhandled CUDA error', 'remote process exited'). The Python wrapper is only the messenger; the real failure is in the NCCL communicator or the network underneath it.","triggerScenarios":"ncclCommInitRank/ncclCommInitRankConfig with mismatched world_size/unique_id across ranks; collectives issued on mismatched counts/dtypes/devices across ranks; a peer rank dying mid-collective (ncclRemoteError); IB/RDMA socket misconfiguration causing init failures (ncclSystemError).","commonSituations":"NCCL_SOCKET_IFNAME pointing at the wrong interface in containers; one worker OOM-crashing so survivors hang then fail; mixed NCCL versions across container images; firewall blocking the NCCL port range; mismatched tensor shapes across ranks in a custom collective.","solutions":["Re-run with NCCL_DEBUG=INFO (or TRACE) and read the first upstream error line above the Python traceback","Verify every rank passes the same world_size, rank and unique_id and issues identical collective shapes/dtypes","Check container network: set NCCL_SOCKET_IFNAME correctly, or increase shm (--ipc=host / --shm-size), and confirm IB devices are visible","Align the NCCL version across all nodes/containers (python -c 'import torch; torch.cuda.nccl.version()') and update CUDA drivers if ncclUnhandledCudaError appears"],"exampleFix":"# before\npython -m vllm.entrypoints.openai.api_server ...  # opaque 'NCCL error: unhandled system error'\n\n# after\nexport NCCL_DEBUG=INFO\npython -m vllm.entrypoints.openai.api_server ...  # traceback now paired with root-cause NCCL log","handlingStrategy":"retry","validationCode":"import subprocess\nfree = int(subprocess.check_output(['df', '-B1', '/dev/shm']).splitlines()[-1].split()[3])\nassert free > required_bytes, f\"/dev/shm too small for NCCL shared buffers\"","typeGuard":null,"tryCatchPattern":"try:\n    dist.all_reduce(x, group=g)\nexcept RuntimeError as e:\n    if \"NCCL error\" in str(e):\n        log.exception(\"NCCL failure; enable NCCL_DEBUG=INFO, check peer liveness\")\n        raise  # NCCL communicators are usually unusable after failure; do not blind-retry","preventionTips":["Always develop with NCCL_DEBUG=INFO","Prefer NCCL communicators (ncclCommInitRankConfig) and destroy cleanly","Align NCCL/CUDA versions across all ranks","Monitor rank liveness so peers fail fast instead of hanging"],"tags":["nccl","distributed","network","multi-node"],"backgroundTag":null,"analyzedSha":"c794754062d49a8fdb63ab3c5215b488b865030c","analyzedAt":"2026-08-14T21:17:39.825Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}