{"record":{"id":"0cc7ff80898d0c2f","repo":"sgl-project/sglang","slug":"dsa-indexer-weights-proj-lora-is-incompatible-with","errorCode":null,"errorMessage":"DSA indexer weights_proj LoRA is incompatible with piecewise/breakable CUDA graph; remove the explicit prefill cuda-graph backend override or drop indexer.weights_proj from the LoRA target modules.","messagePattern":"DSA indexer weights_proj LoRA is incompatible with piecewise/breakable CUDA graph; remove the explicit prefill cuda-graph backend override or drop indexer\\.weights_proj from the LoRA target modules\\.","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/layers/attention/dsa/dsa_indexer.py","lineNumber":1651,"sourceCode":"\n        if (\n            self.use_dsa_indexer_fusion\n            and not in_piecewise_or_breakable_cuda_graph\n            and forward_batch.attn_cp_metadata is None\n        ):\n            q_fp8, weights = self._fused_q_prepare_and_store(\n                x, q_lora, positions, forward_batch, layer_id, act_quant\n            )\n        elif (\n            is_graph_dsa_split_op_surface(forward_batch)\n            and not self.dsa_enable_prefill_cp\n        ):\n            # Default path for non-CP prefill under PCG/BCG: run the whole indexer\n            # (q/k proj, head gate, k-cache store, topk) as a single eager split op\n            # instead of capturing it piecemeal in the graph. The split op is\n            # fusion-aware, so this also covers the fused path here.\n            if weights_proj_lora:\n                raise RuntimeError(GRAPH_WEIGHTS_PROJ_LORA_ERROR)\n            if return_indices:\n                topk_result = torch.full(\n                    (x.shape[0], self.index_topk),\n                    -1,\n                    device=x.device,\n                    dtype=torch.int32,\n                )\n            else:\n                topk_result = torch.empty(\n                    (0, self.index_topk), device=x.device, dtype=torch.int32\n                )\n            graph_dispatch_fn = (\n                bcg_dsa_indexer_prefill_split\n                if is_in_breakable_cuda_graph()\n                else pcg_dsa_indexer_prefill_split\n            )\n            graph_dispatch_fn(\n                layer_id=layer_id,","sourceCodeStart":1633,"sourceCodeEnd":1669,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/layers/attention/dsa/dsa_indexer.py#L1633-L1669","documentation":"Under a piecewise/breakable CUDA graph (PCG/BCG) prefill path, the DSA indexer runs as a single eager split op and cannot apply a LoRA adapter to the weights_proj projection inside that path. If weights_proj is among the LoRA target modules, the indexer raises with GRAPH_WEIGHTS_PROJ_LORA_ERROR instead of silently dropping the adapter.","triggerScenarios":"Serving a DSA model (DeepSeek sparse attention) with a LoRA adapter whose target modules include indexer.weights_proj, while the prefill path runs under a piecewise/breakable CUDA graph (explicit prefill cuda-graph backend override selecting PCG/BCG).","commonSituations":"Applying an indexer-tuned LoRA on DeepSeek-V3.2 with --cuda-graph-backend overrides; LoRA configs that wildcard target_modules (e.g. '.*proj.*') catching indexer.weights_proj unintentionally.","solutions":["Remove indexer.weights_proj from the LoRA target modules list","Drop the explicit prefill cuda-graph backend override so the non-PCG path handles the weights_proj LoRA","If wildcard target_modules are used, exclude the indexer submodule explicitly"],"exampleFix":"# before\n\"target_modules\": [\".*proj.*\"]  # catches indexer.weights_proj under PCG\n# after\n\"target_modules\": [\"q_proj\", \"k_proj\", \"v_proj\", \"o_proj\"]","handlingStrategy":"validation","validationCode":"lora_targets = set(lora_request.target_modules) if lora_request else set()\nif \"indexer.weights_proj\" in lora_targets and using_piecewise_prefill_graph:\n    lora_targets.discard(\"indexer.weights_proj\")  # or drop the graph override","typeGuard":"def lora_compatible_with_pcg(target_modules) -> bool:\n    return \"indexer.weights_proj\" not in target_modules","tryCatchPattern":null,"preventionTips":["Avoid wildcard LoRA target_modules on DSA models","Test each LoRA adapter against the deployed cuda-graph backend in staging"],"tags":["dsa","lora","piecewise-cuda-graph","prefill","deepseek","sglang"],"backgroundTag":"lora-incompatible-with-graph","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}