{"record":{"id":"ca7c34e0ea160c45","repo":"sgl-project/sglang","slug":"unified-kv-dtype-mismatch-kv-unified-kv-dtype-ca7c34","errorCode":null,"errorMessage":"unified_kv dtype mismatch: kv={unified_kv.dtype}, q={q.dtype}","messagePattern":"unified_kv dtype mismatch: kv=(.+?), q=(.+?)","errorType":"validation","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"python/sglang/kernels/ops/attention/dsv4/unified_kv_kernels/paged_prefill.py","lineNumber":237,"sourceCode":"    unified_kv: torch.Tensor,\n    kv_indices_prefix: torch.Tensor,\n    kv_indptr_prefix: torch.Tensor,\n    kv: torch.Tensor,\n    kv_indices_extend: torch.Tensor,\n    kv_indptr_extend: torch.Tensor,\n    attn_sink: torch.Tensor,\n    softmax_scale: float,\n) -> torch.Tensor:\n    if not q.is_cuda:\n        raise RuntimeError(\n            \"Triton sparse_attn_v4_paged_prefill requires CUDA/HIP tensors\"\n        )\n    if q.dtype not in (torch.bfloat16, torch.float16):\n        raise RuntimeError(\n            f\"sparse_attn_v4_paged_prefill expects fp16/bf16 q, got {q.dtype}\"\n        )\n    if unified_kv.dtype != q.dtype:\n        raise RuntimeError(\n            f\"unified_kv dtype mismatch: kv={unified_kv.dtype}, q={q.dtype}\"\n        )\n    if kv.dtype != q.dtype:\n        raise RuntimeError(f\"kv dtype mismatch: kv={kv.dtype}, q={q.dtype}\")\n    if unified_kv.size(-1) != kv.size(-1):\n        raise RuntimeError(\n            f\"head_dim mismatch: unified_kv={unified_kv.size(-1)}, kv={kv.size(-1)}\"\n        )\n\n    T, H, D = q.shape\n    out = torch.empty_like(q)\n    kv_indices_prefix = kv_indices_prefix.to(torch.int32).contiguous()\n    kv_indptr_prefix = kv_indptr_prefix.to(torch.int32).contiguous()\n    kv_indices_extend = kv_indices_extend.to(torch.int32).contiguous()\n    kv_indptr_extend = kv_indptr_extend.to(torch.int32).contiguous()\n\n    block_h = 16  # AMD MFMA min tile\n    block_d = triton.next_power_of_2(D)","sourceCodeStart":219,"sourceCodeEnd":255,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/kernels/ops/attention/dsv4/unified_kv_kernels/paged_prefill.py#L219-L255","documentation":"In sparse_attn_v4_paged_prefill, the paged unified_kv cache must share q's dtype. The kernel does no implicit conversion, so a dtype mismatch aborts the call.","triggerScenarios":"Calling sparse_attn_v4_paged_prefill with q in one 16-bit dtype and unified_kv in the other (fp16 vs bf16) or in fp32.","commonSituations":"KV cache pool allocated with a different default dtype than the model weights; mixed-precision experiments; checkpoint precision differing from server --dtype.","solutions":["Cast/reallocate unified_kv to match q.dtype","Derive the cache dtype from the model config dtype so they cannot diverge","Double-check server --dtype vs cache allocator defaults"],"exampleFix":"// before\nout = sparse_attn_v4_paged_prefill(q_bf16, kv_fp16, ...)\n// after\nout = sparse_attn_v4_paged_prefill(q_bf16, kv_fp16.to(torch.bfloat16), ...)","handlingStrategy":"validation","validationCode":"unified_kv = unified_kv.to(q.dtype)","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Derive cache dtype from model config dtype in one place","Add a preflight check that q, unified_kv, kv all share dtype"],"tags":["attention","dtype","kv-cache","prefill"],"backgroundTag":"tensor-dtype-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}