{"record":{"id":"06c5f0cb28304737","repo":"vllm-project/vllm","slug":"error-happened-when-batch-testing-peer-to-peer-acc","errorCode":null,"errorMessage":"Error happened when batch testing peer-to-peer access from {batch_src} to {batch_tgt}:\n{returned.stderr.decode()}","messagePattern":"Error happened when batch testing peer-to-peer access from (.+?) to (.+?):\n(.+?)","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"vllm/distributed/device_communicators/all_reduce_utils.py","lineNumber":404,"sourceCode":"        # in that case we cannot use spawn method in multiprocessing.\n        # However, `can_actually_p2p` requires spawn method.\n        # The fix is, we use `subprocess` to call the function,\n        # where we have `if __name__ == \"__main__\":` in this file.\n\n        # use a temporary file to store the result\n        # we don't use the output of the subprocess directly,\n        # because the subprocess might produce logging output\n        with tempfile.NamedTemporaryFile() as output_file:\n            input_bytes = pickle.dumps((batch_src, batch_tgt, output_file.name))\n            returned = subprocess.run(\n                [sys.executable, __file__], input=input_bytes, capture_output=True\n            )\n            # check if the subprocess is successful\n            try:\n                returned.check_returncode()\n            except Exception as e:\n                # wrap raised exception to provide more information\n                raise RuntimeError(\n                    f\"Error happened when batch testing \"\n                    f\"peer-to-peer access from {batch_src} to {batch_tgt}:\\n\"\n                    f\"{returned.stderr.decode()}\"\n                ) from e\n            with open(output_file.name, \"rb\") as f:\n                result = pickle.load(f)\n        # Cache entries must be keyed by local indices (0..N-1) because\n        # gpu_p2p_access_check() is called with local ranks.\n        id_to_local = {device_id: local for local, device_id in enumerate(ids)}\n        for _i, _j, r in zip(batch_src, batch_tgt, result):\n            cache[f\"{id_to_local[_i]}->{id_to_local[_j]}\"] = r\n        with open(path, \"w\") as f:\n            json.dump(cache, f, indent=4)\n    if is_distributed:\n        get_world_group().barrier()\n    logger.info(\"reading GPU P2P access cache from %s\", path)\n    with open(path) as f:\n        cache = json.load(f)","sourceCodeStart":386,"sourceCodeEnd":422,"githubUrl":"https://github.com/vllm-project/vllm/blob/c794754062d49a8fdb63ab3c5215b488b865030c/vllm/distributed/device_communicators/all_reduce_utils.py#L386-L422","documentation":"Raised in gpu_p2p_access_check batching (all_reduce_utils.py) when the helper subprocess that tests peer-to-peer GPU access between device pairs exits non-zero. The subprocess re-executes this module with the src/tgt batches pickled on stdin; its stderr is appended so the underlying crash (usually a CUDA init failure in the child) is visible.","triggerScenarios":"vLLM probes P2P CUDA access (multi-GPU startup) and the spawned subprocess dies — e.g. CUDA initialization error in the child, driver/ECC issues, incompatible CUDA_VISIBLE_DEVICES, or OOM during device init.","commonSituations":"Multi-GPU nodes where one GPU is unhealthy (ECC errors, fallen off the bus), a container missing /dev/nvidia* for some GPUs, mixed GPU generations where P2P probing fails, or fork-safety issues causing CUDA init failure in the subprocess.","solutions":["Read the appended stderr — it names the real failure (CUDA error, driver problem) in the child process; fix that first.","Check GPU health (nvidia-smi, dmesg for Xid/ECC) and that all GPUs in CUDA_VISIBLE_DEVICES exist and are usable.","If P2P probing is known-broken on this platform (e.g. some containers or MIG), disable the P2P cache/probe via the documented env (e.g. VLLM_SKIP_P2P_CHECK=1 in versions that support it) or set CUDA_VISIBLE_DEVICES to the healthy subset.","Retry the launch once after fixing the environment; a transient child crash can also come from memory pressure during device init."],"exampleFix":"# before: nvidia-smi shows GPU 2 in ERR/ECC -> subprocess crashes -> RuntimeError\n# after\nnvidia-smi -q -d ECC  # confirm, then\nCUDA_VISIBLE_DEVICES=0,1 vllm serve model -tp 2  # exclude the faulty GPU","handlingStrategy":"try-catch","validationCode":"import subprocess, sys\n\nr = subprocess.run([sys.executable, \"-c\", \"import torch; torch.cuda.init()\"],\n                   capture_output=True)\nif r.returncode != 0:\n    raise SystemExit(f\"CUDA init broken on this node:\\n{r.stderr.decode()}\")","typeGuard":"def node_gpus_healthy() -> bool:\n    import subprocess, sys\n    r = subprocess.run([sys.executable, \"-c\", \"import torch; torch.cuda.init(); print(torch.cuda.device_count())\"], capture_output=True)\n    return r.returncode == 0","tryCatchPattern":"try:\n    gpu_p2p_access_check(...)\nexcept RuntimeError as e:\n    if \"batch testing peer-to-peer\" in str(e):\n        log.error(\"child stderr: %s\", e); run_gpu_diagnostics(); raise","preventionTips":["Surface the embedded child stderr first — it contains the true CUDA failure.","Monitor nvidia-smi/dmesg for ECC and Xid errors on multi-GPU nodes; drain unhealthy GPUs from CUDA_VISIBLE_DEVICES.","For platforms where the P2P probe is known to fail, use the documented skip env or restrict visible devices."],"tags":["vllm","cuda","p2p","multi-gpu","diagnostics"],"backgroundTag":null,"analyzedSha":"c794754062d49a8fdb63ab3c5215b488b865030c","analyzedAt":"2026-08-14T21:17:39.825Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}