sgl-project/sglang · error · RuntimeError

Error happened when batch testing peer-to-peer access from {

Error message

Error happened when batch testing peer-to-peer access from {batch_src} to {batch_tgt}:
{returned.stderr.decode()}

What it means

sglang spawns a helper subprocess to batch-test whether peer-to-peer (P2P) GPU access is possible between groups of source and target GPUs (used by can_p2p for custom all-reduce decisions). If that subprocess exits with a non-zero return code, its stderr is wrapped in this RuntimeError. The root cause is whatever the child process printed to stderr, not the parent logic itself.

Source

Thrown at python/sglang/srt/distributed/device_communicators/custom_all_reduce_utils.py:314

        # in that case we cannot use spawn method in multiprocessing.
        # However, `can_actually_p2p` requires spawn method.
        # The fix is, we use `subprocess` to call the function,
        # where we have `if __name__ == "__main__":` in this file.

        # use a temporary file to store the result
        # we don't use the output of the subprocess directly,
        # because the subprocess might produce logging output
        with tempfile.NamedTemporaryFile() as output_file:
            input_bytes = pickle.dumps((batch_src, batch_tgt, output_file.name))
            returned = subprocess.run(
                [sys.executable, __file__], input=input_bytes, capture_output=True
            )
            # check if the subprocess is successful
            try:
                returned.check_returncode()
            except Exception as e:
                # wrap raised exception to provide more information
                raise RuntimeError(
                    f"Error happened when batch testing "
                    f"peer-to-peer access from {batch_src} to {batch_tgt}:\n"
                    f"{returned.stderr.decode()}"
                ) from e
            with open(output_file.name, "rb") as f:
                result = pickle.load(f)
        for _i, _j, r in zip(batch_src, batch_tgt, result):
            cache[f"{_i}->{_j}"] = r
        with open(path, "w") as f:
            json.dump(cache, f, indent=4)
    if is_distributed:
        get_world_group().barrier()
    logger.info("reading GPU P2P access cache from %s", path)
    with open(path) as f:
        cache = json.load(f)
    _gpu_p2p_access_cache = cache
    return _gpu_p2p_access_cache[f"{src}->{tgt}"]

View on GitHub (pinned to 0132848349)

Solutions

Read the stderr embedded in the exception message — it names the actual child-process failure
Set NCCL_P2P_DISABLE=1 (or rely on sglang's --disable-custom-all-reduce) to skip P2P paths if the topology doesn't support them
Verify both GPUs can enable peer access: run nvidia-smi topo -m and a small torch.cuda.can_device_access_peer check
Fix environment causes: disable MIG, correct CUDA_VISIBLE_DEVICES, update the NVIDIA driver / check BIOS ACS settings

Example fix

# before
p2p = can_p2p(src, tgt)  # raises: Error happened when batch testing peer-to-peer access...

# after
try:
    p2p = can_p2p(src, tgt)
except RuntimeError:
    p2p = False  # fall back to non-P2P all-reduce path

Defensive patterns

Strategy: fallback

Validate before calling

import torch, os
def p2p_plausible(src: int, tgt: int) -> bool:
    if not torch.cuda.is_available():
        return False
    return torch.cuda.can_device_access_peer(src, tgt)

Try / catch

try:
    p2p = can_p2p(src, tgt)
except RuntimeError:
    logger.warning("P2P check failed; falling back to non-P2P all-reduce")
    p2p = False

Prevention

Pre-check with torch.cuda.can_device_access_peer before launching multi-GPU serving
Inspect nvidia-smi topo -m; disable custom all-reduce (NCCL_P2P_DISABLE=1) on PCIe-only or MIG partitions
Ensure consistent CUDA_VISIBLE_DEVICES across ranks so child test processes see valid device ids

When it happens

Trigger: gpu_p2p_access_check(batch_src, batch_tgt) invoking the test subprocess when the child crashes — e.g. CUDA initialization failure in the child, a driver fault while enabling peer access between GPUs, or invalid device ids passed to the test script.

Common situations: Mixing GPUs that lack P2P support over the interconnect (e.g. PCIe-only systems, MIG-enabled devices, GPUs on different NUMA/NCL domains), driver bugs or IOMMU/ACS settings blocking P2P, or CUDA_VISIBLE_DEVICES remapping making the child's device ids invalid.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/b7d84b16011a3c54. Report an issue: GitHub.