sgl-project/sglang · error · RuntimeError

kv_scales must be fp32, got {kv_scales.dtype}

Error message

kv_scales must be fp32, got {kv_scales.dtype}

What it means

For the FP8 group-quantized KV path in sparse_attn_v4_paged_decode, kv_scales must be a float32 tensor so dequantization math is done in fp32 inside the kernel. Other scale dtypes (fp16, bf16, fp64) are rejected.

Source

Thrown at python/sglang/kernels/ops/attention/dsv4/unified_kv_kernels/paged_decode.py:668

    """
    if not q.is_cuda:
        raise RuntimeError(
            "Triton sparse_attn_v4_paged_decode requires CUDA/HIP tensors"
        )
    if q.dtype not in (torch.bfloat16, torch.float16):
        raise RuntimeError(
            f"sparse_attn_v4_paged_decode expects fp16/bf16 q, got {q.dtype}"
        )

    quant_kv = kv_scales is not None
    if quant_kv:
        if unified_kv.dtype != _FP8_DTYPE:
            raise RuntimeError(
                f"kv_scales supplied but unified_kv is {unified_kv.dtype}, "
                f"expected {_FP8_DTYPE}"
            )
        if kv_scales.dtype != torch.float32:
            raise RuntimeError(f"kv_scales must be fp32, got {kv_scales.dtype}")
        D_check = unified_kv.shape[-1]
        if D_check % _FP8_GROUP_SIZE != 0:
            raise RuntimeError(
                f"D={D_check} must be divisible by GROUP_SIZE={_FP8_GROUP_SIZE}"
            )
        expected_g = D_check // _FP8_GROUP_SIZE
        if kv_scales.shape != (unified_kv.shape[0], expected_g):
            raise RuntimeError(
                f"kv_scales shape {tuple(kv_scales.shape)} does not match "
                f"expected ({unified_kv.shape[0]}, {expected_g})"
            )
        if kv_scales.stride(-1) != 1:
            kv_scales = kv_scales.contiguous()
    else:
        if unified_kv.dtype != q.dtype:
            raise RuntimeError(
                f"unified_kv dtype mismatch: kv={unified_kv.dtype}, q={q.dtype}"
            )

View on GitHub (pinned to 0132848349)

Solutions

  1. Cast the scales: kv_scales = kv_scales.float() before the call
  2. Fix the scale-producing quantization routine to emit fp32 from the start

Example fix

// before
scales = scales.to(torch.bfloat16)
// after
scales = scales.to(torch.float32)
Defensive patterns

Strategy: validation

Validate before calling

if kv_scales is not None:
    kv_scales = kv_scales.to(torch.float32)

Type guard

def scales_ok(s: torch.Tensor) -> bool:
    return s.dtype == torch.float32

Prevention

When it happens

Trigger: Calling sparse_attn_v4_paged_decode with kv_scales in a dtype other than torch.float32 while unified_kv is FP8.

Common situations: Scales stored alongside an fp16/bf16 cache and reused after switching to FP8; converting scales to half to save memory; a quantization utility emitting bf16 scales.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/3018da578aab5d5a. Report an issue: GitHub.