{"record":{"id":"63693608a90551e8","repo":"sgl-project/sglang","slug":"kv-scales-supplied-but-unified-kv-is-unified-kv-d","errorCode":null,"errorMessage":"kv_scales supplied but unified_kv is {unified_kv.dtype}, expected {_FP8_DTYPE}","messagePattern":"kv_scales supplied but unified_kv is (.+?), expected (.+?)","errorType":"validation","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"python/sglang/kernels/ops/attention/dsv4/unified_kv_kernels/paged_decode.py","lineNumber":663,"sourceCode":"\n    When ``kv_scales`` is provided, ``unified_kv`` must be e4m3fnuz and\n    ``kv_scales`` must be ``[total_pages, D // GROUP_SIZE]`` fp32 — 1xGROUP_SIZE\n    block-scale quantization. Dequant happens in-kernel; the dot still runs\n    in q.dtype.\n    \"\"\"\n    if not q.is_cuda:\n        raise RuntimeError(\n            \"Triton sparse_attn_v4_paged_decode requires CUDA/HIP tensors\"\n        )\n    if q.dtype not in (torch.bfloat16, torch.float16):\n        raise RuntimeError(\n            f\"sparse_attn_v4_paged_decode expects fp16/bf16 q, got {q.dtype}\"\n        )\n\n    quant_kv = kv_scales is not None\n    if quant_kv:\n        if unified_kv.dtype != _FP8_DTYPE:\n            raise RuntimeError(\n                f\"kv_scales supplied but unified_kv is {unified_kv.dtype}, \"\n                f\"expected {_FP8_DTYPE}\"\n            )\n        if kv_scales.dtype != torch.float32:\n            raise RuntimeError(f\"kv_scales must be fp32, got {kv_scales.dtype}\")\n        D_check = unified_kv.shape[-1]\n        if D_check % _FP8_GROUP_SIZE != 0:\n            raise RuntimeError(\n                f\"D={D_check} must be divisible by GROUP_SIZE={_FP8_GROUP_SIZE}\"\n            )\n        expected_g = D_check // _FP8_GROUP_SIZE\n        if kv_scales.shape != (unified_kv.shape[0], expected_g):\n            raise RuntimeError(\n                f\"kv_scales shape {tuple(kv_scales.shape)} does not match \"\n                f\"expected ({unified_kv.shape[0]}, {expected_g})\"\n            )\n        if kv_scales.stride(-1) != 1:\n            kv_scales = kv_scales.contiguous()","sourceCodeStart":645,"sourceCodeEnd":681,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/kernels/ops/attention/dsv4/unified_kv_kernels/paged_decode.py#L645-L681","documentation":"When kv_scales is supplied to sparse_attn_v4_paged_decode, the kernel expects the unified_kv cache to be FP8-quantized (the _FP8_DTYPE constant). Passing scales together with a non-FP8 (e.g. bf16) KV cache means the quantization contract is broken, so the kernel refuses to run.","triggerScenarios":"Calling sparse_attn_v4_paged_decode(kv_scales=..., unified_kv=<bf16 tensor>) — scales provided but the cache was never quantized to FP8, or the wrong cache tensor was passed.","commonSituations":"Enabling FP8 KV-cache quantization in config but the cache allocator still produced a bf16 pool; mixing quantized and non-quantized paths; stale cache from a previous non-quantized run.","solutions":["If using FP8 KV cache, ensure unified_kv is allocated/cast to the FP8 dtype the kernel expects (e.g. torch.float8_e4m3fn)","If you do not want quantized KV, pass kv_scales=None","Verify the cache allocation code path matches the kv_scales decision (same config flag drives both)"],"exampleFix":"// before\nout = sparse_attn_v4_paged_decode(q, kv_bf16, kv_scales=scales)\n// after\nout = sparse_attn_v4_paged_decode(q, kv_fp8, kv_scales=scales)  # kv quantized to _FP8_DTYPE","handlingStrategy":"validation","validationCode":"use_quant = kv_scales is not None\nif use_quant:\n    assert unified_kv.dtype == torch.float8_e4m3fn, \"kv_scales requires an FP8 unified_kv\"","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Derive the quantized flag from the cache dtype, not from an independent config that can drift","Allocate scales and quantize the cache in the same code path"],"tags":["attention","fp8","quantization","kv-cache","triton"],"backgroundTag":"quantized-kv-dtype-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}