vllm-project/vllm · error · RuntimeError

Error happened when batch testing peer-to-peer access from {

Error message

Error happened when batch testing peer-to-peer access from {batch_src} to {batch_tgt}:
{returned.stderr.decode()}

What it means

Raised in gpu_p2p_access_check batching (all_reduce_utils.py) when the helper subprocess that tests peer-to-peer GPU access between device pairs exits non-zero. The subprocess re-executes this module with the src/tgt batches pickled on stdin; its stderr is appended so the underlying crash (usually a CUDA init failure in the child) is visible.

Source

Thrown at vllm/distributed/device_communicators/all_reduce_utils.py:404

        # in that case we cannot use spawn method in multiprocessing.
        # However, `can_actually_p2p` requires spawn method.
        # The fix is, we use `subprocess` to call the function,
        # where we have `if __name__ == "__main__":` in this file.

        # use a temporary file to store the result
        # we don't use the output of the subprocess directly,
        # because the subprocess might produce logging output
        with tempfile.NamedTemporaryFile() as output_file:
            input_bytes = pickle.dumps((batch_src, batch_tgt, output_file.name))
            returned = subprocess.run(
                [sys.executable, __file__], input=input_bytes, capture_output=True
            )
            # check if the subprocess is successful
            try:
                returned.check_returncode()
            except Exception as e:
                # wrap raised exception to provide more information
                raise RuntimeError(
                    f"Error happened when batch testing "
                    f"peer-to-peer access from {batch_src} to {batch_tgt}:\n"
                    f"{returned.stderr.decode()}"
                ) from e
            with open(output_file.name, "rb") as f:
                result = pickle.load(f)
        # Cache entries must be keyed by local indices (0..N-1) because
        # gpu_p2p_access_check() is called with local ranks.
        id_to_local = {device_id: local for local, device_id in enumerate(ids)}
        for _i, _j, r in zip(batch_src, batch_tgt, result):
            cache[f"{id_to_local[_i]}->{id_to_local[_j]}"] = r
        with open(path, "w") as f:
            json.dump(cache, f, indent=4)
    if is_distributed:
        get_world_group().barrier()
    logger.info("reading GPU P2P access cache from %s", path)
    with open(path) as f:
        cache = json.load(f)

View on GitHub (pinned to c794754062)

Solutions

1. Read the appended stderr — it names the real failure (CUDA error, driver problem) in the child process; fix that first.
2. Check GPU health (nvidia-smi, dmesg for Xid/ECC) and that all GPUs in CUDA_VISIBLE_DEVICES exist and are usable.
3. If P2P probing is known-broken on this platform (e.g. some containers or MIG), disable the P2P cache/probe via the documented env (e.g. VLLM_SKIP_P2P_CHECK=1 in versions that support it) or set CUDA_VISIBLE_DEVICES to the healthy subset.
4. Retry the launch once after fixing the environment; a transient child crash can also come from memory pressure during device init.

Example fix

# before: nvidia-smi shows GPU 2 in ERR/ECC -> subprocess crashes -> RuntimeError
# after
nvidia-smi -q -d ECC  # confirm, then
CUDA_VISIBLE_DEVICES=0,1 vllm serve model -tp 2  # exclude the faulty GPU

import subprocess, sys

r = subprocess.run([sys.executable, "-c", "import torch; torch.cuda.init()"],
                   capture_output=True)
if r.returncode != 0:
    raise SystemExit(f"CUDA init broken on this node:\n{r.stderr.decode()}")

def node_gpus_healthy() -> bool:
    import subprocess, sys
    r = subprocess.run([sys.executable, "-c", "import torch; torch.cuda.init(); print(torch.cuda.device_count())"], capture_output=True)
    return r.returncode == 0

try:
    gpu_p2p_access_check(...)
except RuntimeError as e:
    if "batch testing peer-to-peer" in str(e):
        log.error("child stderr: %s", e); run_gpu_diagnostics(); raise

When it happens

Trigger: vLLM probes P2P CUDA access (multi-GPU startup) and the spawned subprocess dies — e.g. CUDA initialization error in the child, driver/ECC issues, incompatible CUDA_VISIBLE_DEVICES, or OOM during device init.

Common situations: Multi-GPU nodes where one GPU is unhealthy (ECC errors, fallen off the bus), a container missing /dev/nvidia* for some GPUs, mixed GPU generations where P2P probing fails, or fork-safety issues causing CUDA init failure in the subprocess.

Related errors

• CUDA graph capturing detected at an inappropriate time. This
• Sleep mode allocator is not available on platform {type(curr
• HTTP request failed: {0}
• JSON error: {0}
• Tokenizer error: {0}

AI-assisted analysis of vllm-project/vllm@c794754062 (2026-08-14). Data as JSON: /api/errors/06c5f0cb28304737. Report an issue: GitHub.