{"record":{"id":"192f6f7df804f40d","repo":"sgl-project/sglang","slug":"sglang-dsa-topk-broadcast-requires-pynccl-during-c","errorCode":null,"errorMessage":"SGLANG_DSA_TOPK_BROADCAST requires PyNCCL during CUDA graph capture.","messagePattern":"SGLANG_DSA_TOPK_BROADCAST requires PyNCCL during CUDA graph capture\\.","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"critical","filePath":"python/sglang/srt/layers/attention/dsa/dsa_indexer.py","lineNumber":159,"sourceCode":"    from sglang.srt.layers.attention.dsa.dsa_prefill_cuda_graph import (\n        logits_head_gate_graph,\n        scale_head_gate_graph,\n    )\n\n    @register_custom_op(mutates_args=[\"topk_indices\"])\n    @register_split_op()\n    def broadcast_indexer_topk_from_rank0_(topk_indices: torch.Tensor) -> None:\n        _broadcast_indexer_topk_from_rank0_impl(topk_indices)\n\n\ndef _broadcast_indexer_topk_from_rank0_impl(topk_indices: torch.Tensor) -> None:\n    group = get_attn_tp_group()\n    if group.world_size == 1:\n        return\n\n    if topk_indices.device.type == \"cuda\" and torch.cuda.is_current_stream_capturing():\n        if group.pynccl_comm is None:\n            raise RuntimeError(\n                \"SGLANG_DSA_TOPK_BROADCAST requires PyNCCL during CUDA graph capture.\"\n            )\n        with group.pynccl_comm.change_state(enable=True):\n            group.pynccl_comm.broadcast(topk_indices, src=0)\n    else:\n        group.broadcast(topk_indices, src=0)\n\n\ndef _broadcast_indexer_topk_from_rank0(\n    topk_indices: Optional[torch.Tensor],\n) -> Optional[torch.Tensor]:\n    # Sync only the finalized indexer output. Internal topk_transform calls can\n    # be chunked differently across ranks, which would make collectives diverge.\n    if topk_indices is None or not envs.SGLANG_DSA_TOPK_BROADCAST.get():\n        return topk_indices\n\n    if is_in_tc_piecewise_cuda_graph():\n        broadcast_indexer_topk_from_rank0_(topk_indices)","sourceCodeStart":141,"sourceCodeEnd":177,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/layers/attention/dsa/dsa_indexer.py#L141-L177","documentation":"When SGLANG_DSA_TOPK_BROADCAST broadcasts DSA top-k indices from rank 0 during CUDA graph capture, it must use PyNCCL (the graph-capturable NCCL path). If the attention-TP group has no pynccl_comm, broadcasting inside a captured graph is impossible and the indexer raises.","triggerScenarios":"_broadcast_indexer_topk_from_rank0_impl running with attn TP world_size > 1, topk_indices on CUDA, torch.cuda.is_current_stream_capturing() true, and get_attn_tp_group().pynccl_comm is None — i.e. graph capture started without the PyNCCL communicator initialized.","commonSituations":"Disabling the custom allreduce/NCCL path (--disable-custom-all-reduce plus missing pynccl init), startup ordering regressions where graph capture happens before pynccl_comm setup, or flags that skip NCCL communicator creation on single-node TP runs with DSA models.","solutions":["Ensure PyNCCL is enabled/initialized before CUDA graph capture (default init path; avoid flags that disable it)","Initialize/attach the pynccl communicator on the attn TP group before capture (e.g. run one eager warmup forward so communicators are created)","Disable SGLANG_DSA_TOPK_BROADCAST if you don't need rank-0 topk broadcast"],"exampleFix":null,"handlingStrategy":"validation","validationCode":"from sglang.srt.distributed import get_attn_tp_group\ng = get_attn_tp_group()\nif g.world_size > 1 and g.pynccl_comm is None and os.environ.get(\"SGLANG_DSA_TOPK_BROADCAST\"):\n    # run one eager step to init communicators before capture\n    model_runner.warmup_eager()","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Always run an eager warmup forward before CUDA graph capture so NCCL/PyNCCL communicators exist","Avoid flags that disable NCCL communicator initialization when using DSA broadcast on TP > 1"],"tags":["dsa","pynccl","cuda-graph","broadcast","tp","env-var","sglang"],"backgroundTag":"communicator-unavailable-during-graph-capture","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}