{"record":{"id":"3d9fa6d9638d5fca","repo":"sgl-project/sglang","slug":"kv-d-qk-must-match-q-d-qk-d-qk-got-kv-d-qk","errorCode":null,"errorMessage":"kv d_qk must match q d_qk={d_qk}, got {kv_d_qk}","messagePattern":"kv d_qk must match q d_qk=(.+?), got (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/kernels/ops/attention/sparse_mla_q8kv8_prefill_sm90.py","lineNumber":343,"sourceCode":"    if indices.device != device:\n        raise ValueError(\n            f\"indices must be on q's device {device}, got {indices.device}\"\n        )\n\n    if q.dtype != torch.float8_e4m3fn:\n        raise ValueError(f\"q must be torch.float8_e4m3fn, got {q.dtype}\")\n    if kv.dtype != torch.float8_e4m3fn:\n        raise ValueError(f\"kv must be torch.float8_e4m3fn, got {kv.dtype}\")\n\n    if not q.is_contiguous():\n        raise ValueError(\"q must be contiguous\")\n    if not kv.is_contiguous():\n        raise ValueError(\"kv must be contiguous\")\n    if not indices.is_contiguous():\n        raise ValueError(\"indices must be contiguous\")\n\n    if kv_d_qk != d_qk:\n        raise ValueError(f\"kv d_qk must match q d_qk={d_qk}, got {kv_d_qk}\")\n\n    # The CUDA implementation uses B_H=64 and launches h_q / B_H CTAs.\n    # Reject unpadded TP-local head counts instead of launching zero CTAs and\n    # returning uninitialized outputs, which can appear to callers as a hang or\n    # a later collective failure.\n    if h_q == 0 or h_q % 64 != 0:\n        raise ValueError(\n            \"sparse_mla_q8kv8_prefill_fwd requires h_q padded to a positive \"\n            f\"multiple of 64, got {h_q}\"\n        )\n\n    if h_kv != 1:\n        raise ValueError(f\"sparse_mla_q8kv8_prefill_fwd requires h_kv=1, got {h_kv}\")\n\n    if d_qk not in (512, 576):\n        raise ValueError(\n            f\"sparse_mla_q8kv8_prefill_fwd supports d_qk=512/576, got {d_qk}\"\n        )","sourceCodeStart":325,"sourceCodeEnd":361,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/kernels/ops/attention/sparse_mla_q8kv8_prefill_sm90.py#L325-L361","documentation":"The QK head dimension of the kv cache (kv_d_qk, derived from kv's shape) must equal the q tensor's head dimension d_qk (512 or 576). The kernel assumes q and kv share the same per-head QK width for its FP8 dot products.","triggerScenarios":"q has d_qk=576 (e.g. 512+64 RoPE carriers) while the kv cache was allocated with d_qk=512, or vice versa.","commonSituations":"Model variants (DeepSeek MLA with/without rope delta), TP sharding that halves head dims inconsistently, cache allocation using a different config than the attention backend reads.","solutions":["Make the kv cache's QK head dim match q's (reallocate the cache with the correct d_qk)","Verify the model config's qk_head_dim / kv_lora_rank settings flow into both the projection and the cache allocator","If intentional mismatch (nope-only kv), this kernel does not support it; use a different backend"],"exampleFix":"// before\nkv = torch.empty(..., 512, ..., dtype=torch.float8_e4m3fn)  # q is 576\n// after\nkv = torch.empty(..., 576, ..., dtype=torch.float8_e4m3fn)  # matches q d_qk","handlingStrategy":"validation","validationCode":"d_qk = q.shape[-1]\nassert kv.shape[-1] == d_qk, f\"kv d_qk {kv.shape[-1]} != q d_qk {d_qk}\"","typeGuard":"def qkv_dims_match(q: torch.Tensor, kv: torch.Tensor) -> bool:\n    return q.shape[-1] == kv.shape[-1]","tryCatchPattern":null,"preventionTips":["Derive cache dims from the same config object as the attention backend","Add a startup sanity check comparing cache shape to model head dims"],"tags":["shape-mismatch","kv-cache","sparse-attention"],"backgroundTag":"tensor-shape-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}