{"record":{"id":"b7d84b16011a3c54","repo":"sgl-project/sglang","slug":"error-happened-when-batch-testing-peer-to-peer-acc","errorCode":null,"errorMessage":"Error happened when batch testing peer-to-peer access from {batch_src} to {batch_tgt}:\n{returned.stderr.decode()}","messagePattern":"Error happened when batch testing peer-to-peer access from (.+?) to (.+?):\n(.+?)","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/distributed/device_communicators/custom_all_reduce_utils.py","lineNumber":314,"sourceCode":"        # in that case we cannot use spawn method in multiprocessing.\n        # However, `can_actually_p2p` requires spawn method.\n        # The fix is, we use `subprocess` to call the function,\n        # where we have `if __name__ == \"__main__\":` in this file.\n\n        # use a temporary file to store the result\n        # we don't use the output of the subprocess directly,\n        # because the subprocess might produce logging output\n        with tempfile.NamedTemporaryFile() as output_file:\n            input_bytes = pickle.dumps((batch_src, batch_tgt, output_file.name))\n            returned = subprocess.run(\n                [sys.executable, __file__], input=input_bytes, capture_output=True\n            )\n            # check if the subprocess is successful\n            try:\n                returned.check_returncode()\n            except Exception as e:\n                # wrap raised exception to provide more information\n                raise RuntimeError(\n                    f\"Error happened when batch testing \"\n                    f\"peer-to-peer access from {batch_src} to {batch_tgt}:\\n\"\n                    f\"{returned.stderr.decode()}\"\n                ) from e\n            with open(output_file.name, \"rb\") as f:\n                result = pickle.load(f)\n        for _i, _j, r in zip(batch_src, batch_tgt, result):\n            cache[f\"{_i}->{_j}\"] = r\n        with open(path, \"w\") as f:\n            json.dump(cache, f, indent=4)\n    if is_distributed:\n        get_world_group().barrier()\n    logger.info(\"reading GPU P2P access cache from %s\", path)\n    with open(path) as f:\n        cache = json.load(f)\n    _gpu_p2p_access_cache = cache\n    return _gpu_p2p_access_cache[f\"{src}->{tgt}\"]\n","sourceCodeStart":296,"sourceCodeEnd":332,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/distributed/device_communicators/custom_all_reduce_utils.py#L296-L332","documentation":"sglang spawns a helper subprocess to batch-test whether peer-to-peer (P2P) GPU access is possible between groups of source and target GPUs (used by can_p2p for custom all-reduce decisions). If that subprocess exits with a non-zero return code, its stderr is wrapped in this RuntimeError. The root cause is whatever the child process printed to stderr, not the parent logic itself.","triggerScenarios":"gpu_p2p_access_check(batch_src, batch_tgt) invoking the test subprocess when the child crashes — e.g. CUDA initialization failure in the child, a driver fault while enabling peer access between GPUs, or invalid device ids passed to the test script.","commonSituations":"Mixing GPUs that lack P2P support over the interconnect (e.g. PCIe-only systems, MIG-enabled devices, GPUs on different NUMA/NCL domains), driver bugs or IOMMU/ACS settings blocking P2P, or CUDA_VISIBLE_DEVICES remapping making the child's device ids invalid.","solutions":["Read the stderr embedded in the exception message — it names the actual child-process failure","Set NCCL_P2P_DISABLE=1 (or rely on sglang's --disable-custom-all-reduce) to skip P2P paths if the topology doesn't support them","Verify both GPUs can enable peer access: run nvidia-smi topo -m and a small torch.cuda.can_device_access_peer check","Fix environment causes: disable MIG, correct CUDA_VISIBLE_DEVICES, update the NVIDIA driver / check BIOS ACS settings"],"exampleFix":"# before\np2p = can_p2p(src, tgt)  # raises: Error happened when batch testing peer-to-peer access...\n\n# after\ntry:\n    p2p = can_p2p(src, tgt)\nexcept RuntimeError:\n    p2p = False  # fall back to non-P2P all-reduce path","handlingStrategy":"fallback","validationCode":"import torch, os\ndef p2p_plausible(src: int, tgt: int) -> bool:\n    if not torch.cuda.is_available():\n        return False\n    return torch.cuda.can_device_access_peer(src, tgt)","typeGuard":null,"tryCatchPattern":"try:\n    p2p = can_p2p(src, tgt)\nexcept RuntimeError:\n    logger.warning(\"P2P check failed; falling back to non-P2P all-reduce\")\n    p2p = False","preventionTips":["Pre-check with torch.cuda.can_device_access_peer before launching multi-GPU serving","Inspect nvidia-smi topo -m; disable custom all-reduce (NCCL_P2P_DISABLE=1) on PCIe-only or MIG partitions","Ensure consistent CUDA_VISIBLE_DEVICES across ranks so child test processes see valid device ids"],"tags":["cuda","p2p","custom-all-reduce","nccl","subprocess"],"backgroundTag":"gpu-peer-access-failed","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}