{"record":{"id":"3018da578aab5d5a","repo":"sgl-project/sglang","slug":"kv-scales-must-be-fp32-got-kv-scales-dtype","errorCode":null,"errorMessage":"kv_scales must be fp32, got {kv_scales.dtype}","messagePattern":"kv_scales must be fp32, got (.+?)","errorType":"validation","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"python/sglang/kernels/ops/attention/dsv4/unified_kv_kernels/paged_decode.py","lineNumber":668,"sourceCode":"    \"\"\"\n    if not q.is_cuda:\n        raise RuntimeError(\n            \"Triton sparse_attn_v4_paged_decode requires CUDA/HIP tensors\"\n        )\n    if q.dtype not in (torch.bfloat16, torch.float16):\n        raise RuntimeError(\n            f\"sparse_attn_v4_paged_decode expects fp16/bf16 q, got {q.dtype}\"\n        )\n\n    quant_kv = kv_scales is not None\n    if quant_kv:\n        if unified_kv.dtype != _FP8_DTYPE:\n            raise RuntimeError(\n                f\"kv_scales supplied but unified_kv is {unified_kv.dtype}, \"\n                f\"expected {_FP8_DTYPE}\"\n            )\n        if kv_scales.dtype != torch.float32:\n            raise RuntimeError(f\"kv_scales must be fp32, got {kv_scales.dtype}\")\n        D_check = unified_kv.shape[-1]\n        if D_check % _FP8_GROUP_SIZE != 0:\n            raise RuntimeError(\n                f\"D={D_check} must be divisible by GROUP_SIZE={_FP8_GROUP_SIZE}\"\n            )\n        expected_g = D_check // _FP8_GROUP_SIZE\n        if kv_scales.shape != (unified_kv.shape[0], expected_g):\n            raise RuntimeError(\n                f\"kv_scales shape {tuple(kv_scales.shape)} does not match \"\n                f\"expected ({unified_kv.shape[0]}, {expected_g})\"\n            )\n        if kv_scales.stride(-1) != 1:\n            kv_scales = kv_scales.contiguous()\n    else:\n        if unified_kv.dtype != q.dtype:\n            raise RuntimeError(\n                f\"unified_kv dtype mismatch: kv={unified_kv.dtype}, q={q.dtype}\"\n            )","sourceCodeStart":650,"sourceCodeEnd":686,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/kernels/ops/attention/dsv4/unified_kv_kernels/paged_decode.py#L650-L686","documentation":"For the FP8 group-quantized KV path in sparse_attn_v4_paged_decode, kv_scales must be a float32 tensor so dequantization math is done in fp32 inside the kernel. Other scale dtypes (fp16, bf16, fp64) are rejected.","triggerScenarios":"Calling sparse_attn_v4_paged_decode with kv_scales in a dtype other than torch.float32 while unified_kv is FP8.","commonSituations":"Scales stored alongside an fp16/bf16 cache and reused after switching to FP8; converting scales to half to save memory; a quantization utility emitting bf16 scales.","solutions":["Cast the scales: kv_scales = kv_scales.float() before the call","Fix the scale-producing quantization routine to emit fp32 from the start"],"exampleFix":"// before\nscales = scales.to(torch.bfloat16)\n// after\nscales = scales.to(torch.float32)","handlingStrategy":"validation","validationCode":"if kv_scales is not None:\n    kv_scales = kv_scales.to(torch.float32)","typeGuard":"def scales_ok(s: torch.Tensor) -> bool:\n    return s.dtype == torch.float32","tryCatchPattern":null,"preventionTips":["Always emit FP8 group scales as fp32 in your quantization utility","Never downcast scales to save memory"],"tags":["fp8","kv-cache","dtype","scales"],"backgroundTag":"tensor-dtype-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}