{"record":{"id":"9f78488afce85a6d","repo":"sgl-project/sglang","slug":"flashinfer-kda-verify-kernel-only-supports-topk-1","errorCode":null,"errorMessage":"FlashInfer KDA verify kernel only supports topk=1 (retrieve_parent_token must be None).","messagePattern":"FlashInfer KDA verify kernel only supports topk=1 \\(retrieve_parent_token must be None\\)\\.","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/layers/attention/linear/kernels/kda_flashinfer.py","lineNumber":241,"sourceCode":"        dt_bias: torch.Tensor,\n        q: torch.Tensor,\n        k: torch.Tensor,\n        v: torch.Tensor,\n        a: torch.Tensor,\n        b: torch.Tensor,\n        *,\n        ssm_states: torch.Tensor,\n        cache_indices: torch.Tensor,\n        query_start_loc: torch.Tensor,\n        intermediate_states_buffer: torch.Tensor,\n        intermediate_state_indices: torch.Tensor,\n        cache_steps: int,\n        retrieve_parent_token: torch.Tensor,\n        lower_bound: Optional[float] = None,\n        **kwargs,\n    ) -> torch.Tensor:\n        if retrieve_parent_token is not None:\n            raise RuntimeError(\n                \"FlashInfer KDA verify kernel only supports topk=1 \"\n                \"(retrieve_parent_token must be None).\"\n            )\n\n        seq_len = q.shape[1]\n        batch_size = query_start_loc.shape[0] - 1\n        draft_token_num = cache_steps  # T = 1 + num_spec_tokens\n        num_spec_tokens = draft_token_num - 1\n        num_heads = q.shape[2]\n        head_k_dim = q.shape[3]\n        num_v_heads = v.shape[2]\n        head_v_dim = v.shape[3]\n\n        # Packed [1, N*T, ...] inputs, cu_seqlens = query_start_loc (draft stride).\n        # recurrent_kda is bf16-only (see decode), so cast every input to bf16.\n        q_fi = q.reshape(1, seq_len, num_heads, head_k_dim).to(torch.bfloat16)\n        k_fi = k.reshape(1, seq_len, num_heads, head_k_dim).to(torch.bfloat16)\n        v_fi = v.reshape(1, seq_len, num_v_heads, head_v_dim).to(torch.bfloat16)","sourceCodeStart":223,"sourceCodeEnd":259,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/layers/attention/linear/kernels/kda_flashinfer.py#L223-L259","documentation":"The FlashInfer KDA target-verify kernel implements chain verification only; it cannot handle draft trees with topk > 1, which are signaled by a non-None retrieve_parent_token tensor.","triggerScenarios":"Running EAGLE speculative decoding with --speculative-eagle-topk > 1 while the KDA verify backend is flashinfer; the tree verify path passes retrieve_parent_token and the kernel rejects it.","commonSituations":"Config copied from a topk>1 EAGLE3 setup applied to a KDA hybrid model routed to the FlashInfer verify kernel.","solutions":["Set --speculative-eagle-topk 1 so retrieve_parent_token stays None","Use a KDA backend whose target_verify supports tree drafts (e.g. the Triton fallback)","Disable speculative decoding on this model"],"exampleFix":"# before\n--speculative-algorithm EAGLE --speculative-eagle-topk 4\n# after\n--speculative-algorithm EAGLE --speculative-eagle-topk 1","handlingStrategy":"validation","validationCode":"if server_args.speculative_eagle_topk > 1 and kda_verify_backend == 'flashinfer':\n    raise SystemExit('flashinfer KDA verify supports topk=1 only; set --speculative-eagle-topk 1')","typeGuard":"null","tryCatchPattern":null,"preventionTips":["Pin --speculative-eagle-topk 1 for KDA hybrid models","Validate spec config against backend constraints in launch scripts"],"tags":["sglang","kda","speculative-decoding","topk","flashinfer"],"backgroundTag":"unsupported-operation","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}