{"record":{"id":"dfccdd39b642f807","repo":"sgl-project/sglang","slug":"sparse-attn-v4-paged-decode-expects-fp16-bf16-q-g","errorCode":null,"errorMessage":"sparse_attn_v4_paged_decode expects fp16/bf16 q, got {q.dtype}","messagePattern":"sparse_attn_v4_paged_decode expects fp16/bf16 q, got (.+?)","errorType":"validation","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"python/sglang/kernels/ops/attention/dsv4/unified_kv_kernels/paged_decode.py","lineNumber":656,"sourceCode":"    block_h: int | None = None,\n    kv_splits: int | None = None,\n    block_k: int | None = None,\n) -> torch.Tensor:\n    \"\"\"V4 sparse decode Triton implementation: split-K with FUSED fast path,\n    exp2 softmax, CG-safe heuristic. ``block_h`` and ``kv_splits`` are\n    escape hatches for benchmarks; production callers pass neither.\n\n    When ``kv_scales`` is provided, ``unified_kv`` must be e4m3fnuz and\n    ``kv_scales`` must be ``[total_pages, D // GROUP_SIZE]`` fp32 — 1xGROUP_SIZE\n    block-scale quantization. Dequant happens in-kernel; the dot still runs\n    in q.dtype.\n    \"\"\"\n    if not q.is_cuda:\n        raise RuntimeError(\n            \"Triton sparse_attn_v4_paged_decode requires CUDA/HIP tensors\"\n        )\n    if q.dtype not in (torch.bfloat16, torch.float16):\n        raise RuntimeError(\n            f\"sparse_attn_v4_paged_decode expects fp16/bf16 q, got {q.dtype}\"\n        )\n\n    quant_kv = kv_scales is not None\n    if quant_kv:\n        if unified_kv.dtype != _FP8_DTYPE:\n            raise RuntimeError(\n                f\"kv_scales supplied but unified_kv is {unified_kv.dtype}, \"\n                f\"expected {_FP8_DTYPE}\"\n            )\n        if kv_scales.dtype != torch.float32:\n            raise RuntimeError(f\"kv_scales must be fp32, got {kv_scales.dtype}\")\n        D_check = unified_kv.shape[-1]\n        if D_check % _FP8_GROUP_SIZE != 0:\n            raise RuntimeError(\n                f\"D={D_check} must be divisible by GROUP_SIZE={_FP8_GROUP_SIZE}\"\n            )\n        expected_g = D_check // _FP8_GROUP_SIZE","sourceCodeStart":638,"sourceCodeEnd":674,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/kernels/ops/attention/dsv4/unified_kv_kernels/paged_decode.py#L638-L674","documentation":"The Triton sparse DeepSeek-V4 paged decode attention kernel only accepts query tensors in torch.float16 or torch.bfloat16. Any other dtype (fp32, fp8, etc.) is rejected up front because the kernel's Triton code is specialized for 16-bit inputs. This mirrors the CPU/CUDA guard immediately above it.","triggerScenarios":"Calling sparse_attn_v4_paged_decode with a q tensor whose dtype is not fp16/bf16, e.g. a model or test that produced float32 queries, or a quantized path that passes fp8 q directly.","commonSituations":"Unit tests building fp32 fixtures; a model variant that leaves q in fp32; accidentally passing a dequantized/upscore tensor; running with a dtype override like --dtype float32.","solutions":["Cast q to bf16/fp16 before calling: q = q.to(torch.bfloat16)","Check the model's dtype configuration (server --dtype) matches a supported 16-bit dtype","If you control the caller, add a dtype assertion early so the failure points at the producer, not the kernel"],"exampleFix":"// before\nout = sparse_attn_v4_paged_decode(q, ...)  # q is float32\n// after\nout = sparse_attn_v4_paged_decode(q.to(torch.bfloat16), ...)","handlingStrategy":"type-guard","validationCode":"assert q.is_cuda and q.dtype in (torch.float16, torch.bfloat16), f\"q must be cuda fp16/bf16, got {q.device} {q.dtype}\"","typeGuard":"def is_q_valid(q: torch.Tensor) -> bool:\n    return q.is_cuda and q.dtype in (torch.float16, torch.bfloat16)","tryCatchPattern":null,"preventionTips":["Standardize one compute dtype (bf16) for the whole model and assert it at entry points","Add dtype/device asserts in test fixtures, not just in the kernel wrapper"],"tags":["attention","dtype","triton","deepseek","gpu"],"backgroundTag":"tensor-dtype-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}