{"record":{"id":"d66e149d3e38775a","repo":"sgl-project/sglang","slug":"unified-kv-dtype-mismatch-kv-unified-kv-dtype","errorCode":null,"errorMessage":"unified_kv dtype mismatch: kv={unified_kv.dtype}, q={q.dtype}","messagePattern":"unified_kv dtype mismatch: kv=(.+?), q=(.+?)","errorType":"validation","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"python/sglang/kernels/ops/attention/dsv4/unified_kv_kernels/paged_decode.py","lineNumber":684,"sourceCode":"            )\n        if kv_scales.dtype != torch.float32:\n            raise RuntimeError(f\"kv_scales must be fp32, got {kv_scales.dtype}\")\n        D_check = unified_kv.shape[-1]\n        if D_check % _FP8_GROUP_SIZE != 0:\n            raise RuntimeError(\n                f\"D={D_check} must be divisible by GROUP_SIZE={_FP8_GROUP_SIZE}\"\n            )\n        expected_g = D_check // _FP8_GROUP_SIZE\n        if kv_scales.shape != (unified_kv.shape[0], expected_g):\n            raise RuntimeError(\n                f\"kv_scales shape {tuple(kv_scales.shape)} does not match \"\n                f\"expected ({unified_kv.shape[0]}, {expected_g})\"\n            )\n        if kv_scales.stride(-1) != 1:\n            kv_scales = kv_scales.contiguous()\n    else:\n        if unified_kv.dtype != q.dtype:\n            raise RuntimeError(\n                f\"unified_kv dtype mismatch: kv={unified_kv.dtype}, q={q.dtype}\"\n            )\n\n    T, H, D = q.shape\n    out = torch.empty_like(q)\n\n    if block_h is None:\n        block_h = triton.next_power_of_2(min(H, 64))\n    else:\n        block_h = triton.next_power_of_2(block_h)\n    block_h = max(block_h, 16)  # AMD MFMA min tile\n\n    n_head_blocks = (H + block_h - 1) // block_h\n    h_padded = n_head_blocks * block_h\n    block_d = triton.next_power_of_2(D)\n\n    if kv_splits is None:\n        kv_splits = _kv_splits_heuristic(T, H, block_h)","sourceCodeStart":666,"sourceCodeEnd":702,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/kernels/ops/attention/dsv4/unified_kv_kernels/paged_decode.py#L666-L702","documentation":"In the non-quantized path (kv_scales=None), sparse_attn_v4_paged_decode requires unified_kv to have the same dtype as q (both fp16 or both bf16). Mixed dtypes would either break the Triton kernel or silently corrupt the attention math.","triggerScenarios":"Calling sparse_attn_v4_paged_decode with, e.g., q in bf16 and unified_kv in fp16, and no kv_scales.","commonSituations":"Model weights cast to bf16 while the KV cache pool was allocated fp16 (or vice versa); --dtype flag inconsistent with the cache allocator default; mixing checkpoints of different precision.","solutions":["Cast unified_kv to q.dtype (or reallocate the cache pool with the model dtype)","Make the KV cache allocator derive its dtype from the model config dtype so they can never diverge","Verify server/model dtype settings are consistent end to end"],"exampleFix":"// before\nout = sparse_attn_v4_paged_decode(q_bf16, kv_fp16)\n// after\nout = sparse_attn_v4_paged_decode(q_bf16, kv_fp16.to(torch.bfloat16))","handlingStrategy":"validation","validationCode":"if kv_scales is None:\n    unified_kv = unified_kv.to(q.dtype)","typeGuard":"def kv_dtype_ok(q, kv):\n    return kv.dtype == q.dtype or (kv.dtype == torch.float8_e4m3fn)","tryCatchPattern":null,"preventionTips":["Allocate the KV pool with the model's compute dtype from config","Add a single dtype-consistency check across q/unified_kv/kv before dispatch"],"tags":["attention","dtype","kv-cache","triton"],"backgroundTag":"tensor-dtype-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}