sgl-project/sglang · error · RuntimeError

Error happened when batch testing peer-to-peer access from

Error message

Error happened when batch testing peer-to-peer access from {batch_src} to {batch_tgt}:
{returned.stderr.decode()}

What it means

sglang spawns a helper subprocess to batch-test whether peer-to-peer (P2P) GPU access is possible between groups of source and target GPUs (used by can_p2p for custom all-reduce decisions). If that subprocess exits with a non-zero return code, its stderr is wrapped in this RuntimeError. The root cause is whatever the child process printed to stderr, not the parent logic itself.

Solutions

Read the stderr embedded in the exception message — it names the actual child-process failure
Set NCCL_P2P_DISABLE=1 (or rely on sglang's --disable-custom-all-reduce) to skip P2P paths if the topology doesn't support them
Verify both GPUs can enable peer access: run nvidia-smi topo -m and a small torch.cuda.can_device_access_peer check
Fix environment causes: disable MIG, correct CUDA_VISIBLE_DEVICES, update the NVIDIA driver / check BIOS ACS settings

Example fix

# before
p2p = can_p2p(src, tgt)  # raises: Error happened when batch testing peer-to-peer access...

# after
try:
    p2p = can_p2p(src, tgt)
except RuntimeError:
    p2p = False  # fall back to non-P2P all-reduce path

Defensive patterns

Strategy: fallback

Validate before calling

import torch, os
def p2p_plausible(src: int, tgt: int) -> bool:
    if not torch.cuda.is_available():
        return False
    return torch.cuda.can_device_access_peer(src, tgt)

Try / catch

try:
    p2p = can_p2p(src, tgt)
except RuntimeError:
    logger.warning("P2P check failed; falling back to non-P2P all-reduce")
    p2p = False

Prevention

Pre-check with torch.cuda.can_device_access_peer before launching multi-GPU serving
Inspect nvidia-smi topo -m; disable custom all-reduce (NCCL_P2P_DISABLE=1) on PCIe-only or MIG partitions
Ensure consistent CUDA_VISIBLE_DEVICES across ranks so child test processes see valid device ids

When it happens

Trigger: gpu_p2p_access_check(batch_src, batch_tgt) invoking the test subprocess when the child crashes — e.g. CUDA initialization failure in the child, a driver fault while enabling peer access between GPUs, or invalid device ids passed to the test script.

Common situations: Mixing GPUs that lack P2P support over the interconnect (e.g. PCIe-only systems, MIG-enabled devices, GPUs on different NUMA/NCL domains), driver bugs or IOMMU/ACS settings blocking P2P, or CUDA_VISIBLE_DEVICES remapping making the child's device ids invalid.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/b7d84b16011a3c54. Report an issue: GitHub.

Appendix: source

Thrown at python/sglang/srt/distributed/device_communicators/custom_all_reduce_utils.py:314

        # in that case we cannot use spawn method in multiprocessing.
        # However, `can_actually_p2p` requires spawn method.
        # The fix is, we use `subprocess` to call the function,
        # where we have `if __name__ == "__main__":` in this file.

        # use a temporary file to store the result
        # we don't use the output of the subprocess directly,
        # because the subprocess might produce logging output
        with tempfile.NamedTemporaryFile() as output_file:
            input_bytes = pickle.dumps((batch_src, batch_tgt, output_file.name))
            returned = subprocess.run(
                [sys.executable, __file__], input=input_bytes, capture_output=True
            )
            # check if the subprocess is successful
            try:
                returned.check_returncode()
            except Exception as e:
                # wrap raised exception to provide more information
                raise RuntimeError(
                    f"Error happened when batch testing "
                    f"peer-to-peer access from {batch_src} to {batch_tgt}:\n"
                    f"{returned.stderr.decode()}"
                ) from e
            with open(output_file.name, "rb") as f:
                result = pickle.load(f)
        for _i, _j, r in zip(batch_src, batch_tgt, result):
            cache[f"{_i}->{_j}"] = r
        with open(path, "w") as f:
            json.dump(cache, f, indent=4)
    if is_distributed:
        get_world_group().barrier()
    logger.info("reading GPU P2P access cache from %s", path)
    with open(path) as f:
        cache = json.load(f)
    _gpu_p2p_access_cache = cache
    return _gpu_p2p_access_cache[f"{src}->{tgt}"]

View on GitHub (pinned to 0132848349)