sgl-project/sglang · error · RuntimeError

FlashInfer KDA verify kernel only supports topk=1 (retrieve_

Error message

FlashInfer KDA verify kernel only supports topk=1 (retrieve_parent_token must be None).

What it means

The FlashInfer KDA target-verify kernel implements chain verification only; it cannot handle draft trees with topk > 1, which are signaled by a non-None retrieve_parent_token tensor.

Source

Thrown at python/sglang/srt/layers/attention/linear/kernels/kda_flashinfer.py:241

        dt_bias: torch.Tensor,
        q: torch.Tensor,
        k: torch.Tensor,
        v: torch.Tensor,
        a: torch.Tensor,
        b: torch.Tensor,
        *,
        ssm_states: torch.Tensor,
        cache_indices: torch.Tensor,
        query_start_loc: torch.Tensor,
        intermediate_states_buffer: torch.Tensor,
        intermediate_state_indices: torch.Tensor,
        cache_steps: int,
        retrieve_parent_token: torch.Tensor,
        lower_bound: Optional[float] = None,
        **kwargs,
    ) -> torch.Tensor:
        if retrieve_parent_token is not None:
            raise RuntimeError(
                "FlashInfer KDA verify kernel only supports topk=1 "
                "(retrieve_parent_token must be None)."
            )

        seq_len = q.shape[1]
        batch_size = query_start_loc.shape[0] - 1
        draft_token_num = cache_steps  # T = 1 + num_spec_tokens
        num_spec_tokens = draft_token_num - 1
        num_heads = q.shape[2]
        head_k_dim = q.shape[3]
        num_v_heads = v.shape[2]
        head_v_dim = v.shape[3]

        # Packed [1, N*T, ...] inputs, cu_seqlens = query_start_loc (draft stride).
        # recurrent_kda is bf16-only (see decode), so cast every input to bf16.
        q_fi = q.reshape(1, seq_len, num_heads, head_k_dim).to(torch.bfloat16)
        k_fi = k.reshape(1, seq_len, num_heads, head_k_dim).to(torch.bfloat16)
        v_fi = v.reshape(1, seq_len, num_v_heads, head_v_dim).to(torch.bfloat16)

View on GitHub (pinned to 0132848349)

Solutions

1. Set --speculative-eagle-topk 1 so retrieve_parent_token stays None
2. Use a KDA backend whose target_verify supports tree drafts (e.g. the Triton fallback)
3. Disable speculative decoding on this model

Example fix

# before
--speculative-algorithm EAGLE --speculative-eagle-topk 4
# after
--speculative-algorithm EAGLE --speculative-eagle-topk 1

if server_args.speculative_eagle_topk > 1 and kda_verify_backend == 'flashinfer':
    raise SystemExit('flashinfer KDA verify supports topk=1 only; set --speculative-eagle-topk 1')

null

When it happens

Trigger: Running EAGLE speculative decoding with --speculative-eagle-topk > 1 while the KDA verify backend is flashinfer; the tree verify path passes retrieve_parent_token and the kernel rejects it.

Common situations: Config copied from a topk>1 EAGLE3 setup applied to a KDA hybrid model routed to the FlashInfer verify kernel.

Understand the failure class

Background: UnsupportedOperationException and "is not supported" errors: when a library deliberately refuses a call — this error's family across 30 libraries.

Related errors

• CuteDSLKDAKernel does not support target_verify
• FlashInfer KDA kernel (recurrent_kda) is not available. Requ
• f"recurrent_kda state pool breaks the compiled stride contra
• f"KDA verify needs {draft_token_num} scratch steps, but inte
• FlashInfer KDA verify requires an identity intermediate row-

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/9f78488afce85a6d. Report an issue: GitHub.