sgl-project/sglang · critical · RuntimeError

--enable-tp-lm-head-all-to-all requires an available PyNCCL

Error message

--enable-tp-lm-head-all-to-all requires an available PyNCCL communicator for CUDA graph capture.

What it means

_prewarm_tp_lm_head_all_to_all runs during init_torch_distributed when --enable-tp-lm-head-all-to-all is set, and requires the TP group's PyNCCL communicator to exist and be available for CUDA-graph-capturable P2P. If pynccl_comm is None or not available (e.g. NCCL P2P disabled or not initialized), it raises RuntimeError at startup.

Source

Thrown at python/sglang/srt/distributed/bootstrap.py:308

    # Single warmup all_reduce to initialize NCCL/RCCL/HCCL communicator
    warmup_tensor = torch.zeros(1, device=torch.cuda.current_device())
    dist.all_reduce(warmup_tensor, group=tp_group_handle)
    current_platform.synchronize()

    warmup_elapsed = time.perf_counter() - warmup_start
    logger.info(
        f"NCCL/RCCL/HCCL warmup completed in {warmup_elapsed:.3f}s "
        f"(tp_size={tp_size}, pp_size={pp_size}, ep_size={moe_ep_size})"
    )


def _prewarm_tp_lm_head_all_to_all() -> None:
    """Materialize PyNCCL P2P resources before model-memory accounting."""
    warmup_start = time.perf_counter()
    tp_group = get_tp_group()
    pynccl_comm = tp_group.pynccl_comm
    if pynccl_comm is None or not pynccl_comm.available:
        raise RuntimeError(
            "--enable-tp-lm-head-all-to-all requires an available PyNCCL "
            "communicator for CUDA graph capture."
        )

    numel = tp_group.world_size * _TP_ALL_TO_ALL_WARMUP_BYTES_PER_PEER
    warmup_input = torch.empty(numel, dtype=torch.uint8, device=tp_group.device)
    warmup_output = torch.empty_like(warmup_input)

    # PyNCCL is disabled outside graph-capture contexts by default. Enable it
    # explicitly so eager startup does not fall back to ProcessGroupNCCL and
    # miss the P2P resources required by the captured all-to-all.
    with pynccl_comm.change_state(enable=True):
        pynccl_comm.all_to_all_single(warmup_output, warmup_input)
    current_platform.synchronize()

    del warmup_input, warmup_output
    current_platform.empty_cache()
    warmup_elapsed = time.perf_counter() - warmup_start

View on GitHub (pinned to 0132848349)

Solutions

1. Ensure GPUs support P2P (unblock it: enable IOMMU/ACS settings, remove NCCL_P2P_DISABLE=1)
2. Verify PyNCCL is installed/importable in the environment and the communicator initialized before this stage
3. Drop --enable-tp-lm-head-all-to-all if the hardware cannot support P2P all-to-all

Example fix

# before
NCCL_P2P_DISABLE=1 python -m sglang.launch_server ... --enable-tp-lm-head-all-to-all
# after
python -m sglang.launch_server ... --enable-tp-lm-head-all-to-all

from sglang.srt.distributed import get_tp_group
c = get_tp_group().pynccl_comm
assert c is not None and c.available, "PyNCCL P2P unavailable; cannot use --enable-tp-lm-head-all-to-all"

When it happens

Trigger: Launching with --enable-tp-lm-head-all-to-all when PyNCCL is unavailable: NCCL not installed/compiled, P2P disabled (NCCL_P2P_DISABLE=1 or platform without P2P, e.g. NVLink absent), or the communicator not yet created at bootstrap time.

Common situations: Enabling the TP LM-head all-to-all optimization on machines without GPU P2P (PCIe topologies, IOMMU/vfio blocking P2P), in containers with CUDA_CAPABILITIES constraints, or with NCCL env vars disabling P2P.

Related errors

• Pi05 action state broadcast returned None
• The memory capacity is unbalanced. Some GPUs may be occupied
• Error happened when batch testing peer-to-peer access from {
• NCCL only supports CUDA, ROCm and MUSA backends.
• VMM handle export failed: FABRIC export failed on at least o

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/36a9e9c9131163de. Report an issue: GitHub.