{"record":{"id":"36a9e9c9131163de","repo":"sgl-project/sglang","slug":"enable-tp-lm-head-all-to-all-requires-an-availab","errorCode":null,"errorMessage":"--enable-tp-lm-head-all-to-all requires an available PyNCCL communicator for CUDA graph capture.","messagePattern":"--enable-tp-lm-head-all-to-all requires an available PyNCCL communicator for CUDA graph capture\\.","errorType":"validation","errorClass":"RuntimeError","httpStatus":null,"severity":"critical","filePath":"python/sglang/srt/distributed/bootstrap.py","lineNumber":308,"sourceCode":"    # Single warmup all_reduce to initialize NCCL/RCCL/HCCL communicator\n    warmup_tensor = torch.zeros(1, device=torch.cuda.current_device())\n    dist.all_reduce(warmup_tensor, group=tp_group_handle)\n    current_platform.synchronize()\n\n    warmup_elapsed = time.perf_counter() - warmup_start\n    logger.info(\n        f\"NCCL/RCCL/HCCL warmup completed in {warmup_elapsed:.3f}s \"\n        f\"(tp_size={tp_size}, pp_size={pp_size}, ep_size={moe_ep_size})\"\n    )\n\n\ndef _prewarm_tp_lm_head_all_to_all() -> None:\n    \"\"\"Materialize PyNCCL P2P resources before model-memory accounting.\"\"\"\n    warmup_start = time.perf_counter()\n    tp_group = get_tp_group()\n    pynccl_comm = tp_group.pynccl_comm\n    if pynccl_comm is None or not pynccl_comm.available:\n        raise RuntimeError(\n            \"--enable-tp-lm-head-all-to-all requires an available PyNCCL \"\n            \"communicator for CUDA graph capture.\"\n        )\n\n    numel = tp_group.world_size * _TP_ALL_TO_ALL_WARMUP_BYTES_PER_PEER\n    warmup_input = torch.empty(numel, dtype=torch.uint8, device=tp_group.device)\n    warmup_output = torch.empty_like(warmup_input)\n\n    # PyNCCL is disabled outside graph-capture contexts by default. Enable it\n    # explicitly so eager startup does not fall back to ProcessGroupNCCL and\n    # miss the P2P resources required by the captured all-to-all.\n    with pynccl_comm.change_state(enable=True):\n        pynccl_comm.all_to_all_single(warmup_output, warmup_input)\n    current_platform.synchronize()\n\n    del warmup_input, warmup_output\n    current_platform.empty_cache()\n    warmup_elapsed = time.perf_counter() - warmup_start","sourceCodeStart":290,"sourceCodeEnd":326,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/distributed/bootstrap.py#L290-L326","documentation":"_prewarm_tp_lm_head_all_to_all runs during init_torch_distributed when --enable-tp-lm-head-all-to-all is set, and requires the TP group's PyNCCL communicator to exist and be available for CUDA-graph-capturable P2P. If pynccl_comm is None or not available (e.g. NCCL P2P disabled or not initialized), it raises RuntimeError at startup.","triggerScenarios":"Launching with --enable-tp-lm-head-all-to-all when PyNCCL is unavailable: NCCL not installed/compiled, P2P disabled (NCCL_P2P_DISABLE=1 or platform without P2P, e.g. NVLink absent), or the communicator not yet created at bootstrap time.","commonSituations":"Enabling the TP LM-head all-to-all optimization on machines without GPU P2P (PCIe topologies, IOMMU/vfio blocking P2P), in containers with CUDA_CAPABILITIES constraints, or with NCCL env vars disabling P2P.","solutions":["Ensure GPUs support P2P (unblock it: enable IOMMU/ACS settings, remove NCCL_P2P_DISABLE=1)","Verify PyNCCL is installed/importable in the environment and the communicator initialized before this stage","Drop --enable-tp-lm-head-all-to-all if the hardware cannot support P2P all-to-all"],"exampleFix":"# before\nNCCL_P2P_DISABLE=1 python -m sglang.launch_server ... --enable-tp-lm-head-all-to-all\n# after\npython -m sglang.launch_server ... --enable-tp-lm-head-all-to-all","handlingStrategy":"validation","validationCode":"from sglang.srt.distributed import get_tp_group\nc = get_tp_group().pynccl_comm\nassert c is not None and c.available, \"PyNCCL P2P unavailable; cannot use --enable-tp-lm-head-all-to-all\"","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Verify nvidia-smi topo shows P2P between GPUs before enabling the flag","Do not set NCCL_P2P_DISABLE=1 with this feature"],"tags":["distributed","nccl","p2p","tp","lm-head"],"backgroundTag":"nccl-communicator-unavailable","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}