{"record":{"id":"54919db54a9458bf","repo":"sgl-project/sglang","slug":"f-kda-verify-needs-draft-token-num-scratch-steps","errorCode":null,"errorMessage":"f\"KDA verify needs {draft_token_num} scratch steps, but intermediate_ssm only has {scratch_steps}.\"","messagePattern":"f\"KDA verify needs (.+?) scratch steps, but intermediate_ssm only has (.+?)\\.\"","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/layers/attention/linear/kernels/kda_flashinfer.py","lineNumber":270,"sourceCode":"        num_v_heads = v.shape[2]\n        head_v_dim = v.shape[3]\n\n        # Packed [1, N*T, ...] inputs, cu_seqlens = query_start_loc (draft stride).\n        # recurrent_kda is bf16-only (see decode), so cast every input to bf16.\n        q_fi = q.reshape(1, seq_len, num_heads, head_k_dim).to(torch.bfloat16)\n        k_fi = k.reshape(1, seq_len, num_heads, head_k_dim).to(torch.bfloat16)\n        v_fi = v.reshape(1, seq_len, num_v_heads, head_v_dim).to(torch.bfloat16)\n        g_fi = a.reshape(1, seq_len, num_v_heads, head_k_dim).to(torch.bfloat16)\n        beta_fi = self._beta_logit_to_prob(b).reshape(1, seq_len, num_v_heads)\n\n        A_log_fi, dt_bias_fi = self._prep_gate_params(A_log, dt_bias)\n\n        # recurrent_kda indexes a flat state pool. Map each request/step to the\n        # matching slot in SGLang's [scratch_row, allocated_step, HV, V, K] buffer.\n        scratch = intermediate_states_buffer  # [N_scratch, T, HV, V, K]\n        scratch_steps = scratch.shape[1]\n        if draft_token_num > scratch_steps:\n            raise RuntimeError(\n                f\"KDA verify needs {draft_token_num} scratch steps, \"\n                f\"but intermediate_ssm only has {scratch_steps}.\"\n            )\n\n        base_rows = intermediate_state_indices[:batch_size]\n        cache_key = (\n            id(intermediate_state_indices),\n            batch_size,\n            draft_token_num,\n            scratch_steps,\n        )\n        ssm_state_indices = self._verify_idx_cache.get(cache_key)\n        if ssm_state_indices is None:\n            # The fast seed copy below assumes row n in scratch belongs to request n.\n            expected = torch.arange(\n                batch_size, device=base_rows.device, dtype=base_rows.dtype\n            )\n            if not torch.equal(base_rows, expected):","sourceCodeStart":252,"sourceCodeEnd":288,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/layers/attention/linear/kernels/kda_flashinfer.py#L252-L288","documentation":"During target_verify, each of the draft_token_num speculative steps needs one scratch row in the intermediate SSM buffer; if the buffer's step dimension T is smaller than draft_token_num, the kernel cannot index the flat state pool and raises RuntimeError.","triggerScenarios":"Speculative config produces more draft tokens per request than the intermediate_ssm buffer was allocated for (e.g. speculative_num_steps increased after cache sizing, or draft_token_num > buffer T).","commonSituations":"Raising --speculative-num-steps / --speculative-eagle-topk without resizing the mamba intermediate buffer; mismatched server args between cache allocation and verify path.","solutions":["Increase the intermediate SSM buffer's step dimension to at least the max draft_token_num (align speculative_num_steps with cache sizing)","Lower --speculative-num-steps so draft_token_num fits the buffer","Re-run with the standard speculative config used when the cache was allocated"],"exampleFix":"# before\n--speculative-num-steps 8   # buffer T was sized for 4\n# after\n--speculative-num-steps 4   # or resize intermediate_ssm to T>=8","handlingStrategy":"validation","validationCode":"draft_token_num = speculative_num_steps  # per request\nassert draft_token_num <= intermediate_ssm.shape[1], (\n    f'need {draft_token_num} scratch steps, buffer has {intermediate_ssm.shape[1]}')","typeGuard":"null","tryCatchPattern":null,"preventionTips":["Size the intermediate SSM buffer's step dim to the max draft_token_num at startup","Keep speculative_num_steps consistent with cache allocation args"],"tags":["sglang","kda","speculative-decoding","buffer-sizing"],"backgroundTag":"buffer-capacity-exceeded","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}