sgl-project/sglang · error · RuntimeError

kv_scales supplied but unified_kv is {unified_kv.dtype}, exp

Error message

kv_scales supplied but unified_kv is {unified_kv.dtype}, expected {_FP8_DTYPE}

What it means

When kv_scales is supplied to sparse_attn_v4_paged_decode, the kernel expects the unified_kv cache to be FP8-quantized (the _FP8_DTYPE constant). Passing scales together with a non-FP8 (e.g. bf16) KV cache means the quantization contract is broken, so the kernel refuses to run.

Source

Thrown at python/sglang/kernels/ops/attention/dsv4/unified_kv_kernels/paged_decode.py:663

    When ``kv_scales`` is provided, ``unified_kv`` must be e4m3fnuz and
    ``kv_scales`` must be ``[total_pages, D // GROUP_SIZE]`` fp32 — 1xGROUP_SIZE
    block-scale quantization. Dequant happens in-kernel; the dot still runs
    in q.dtype.
    """
    if not q.is_cuda:
        raise RuntimeError(
            "Triton sparse_attn_v4_paged_decode requires CUDA/HIP tensors"
        )
    if q.dtype not in (torch.bfloat16, torch.float16):
        raise RuntimeError(
            f"sparse_attn_v4_paged_decode expects fp16/bf16 q, got {q.dtype}"
        )

    quant_kv = kv_scales is not None
    if quant_kv:
        if unified_kv.dtype != _FP8_DTYPE:
            raise RuntimeError(
                f"kv_scales supplied but unified_kv is {unified_kv.dtype}, "
                f"expected {_FP8_DTYPE}"
            )
        if kv_scales.dtype != torch.float32:
            raise RuntimeError(f"kv_scales must be fp32, got {kv_scales.dtype}")
        D_check = unified_kv.shape[-1]
        if D_check % _FP8_GROUP_SIZE != 0:
            raise RuntimeError(
                f"D={D_check} must be divisible by GROUP_SIZE={_FP8_GROUP_SIZE}"
            )
        expected_g = D_check // _FP8_GROUP_SIZE
        if kv_scales.shape != (unified_kv.shape[0], expected_g):
            raise RuntimeError(
                f"kv_scales shape {tuple(kv_scales.shape)} does not match "
                f"expected ({unified_kv.shape[0]}, {expected_g})"
            )
        if kv_scales.stride(-1) != 1:
            kv_scales = kv_scales.contiguous()

View on GitHub (pinned to 0132848349)

Solutions

1. If using FP8 KV cache, ensure unified_kv is allocated/cast to the FP8 dtype the kernel expects (e.g. torch.float8_e4m3fn)
2. If you do not want quantized KV, pass kv_scales=None
3. Verify the cache allocation code path matches the kv_scales decision (same config flag drives both)

Example fix

// before
out = sparse_attn_v4_paged_decode(q, kv_bf16, kv_scales=scales)
// after
out = sparse_attn_v4_paged_decode(q, kv_fp8, kv_scales=scales)  # kv quantized to _FP8_DTYPE

use_quant = kv_scales is not None
if use_quant:
    assert unified_kv.dtype == torch.float8_e4m3fn, "kv_scales requires an FP8 unified_kv"

When it happens

Trigger: Calling sparse_attn_v4_paged_decode(kv_scales=..., unified_kv=<bf16 tensor>) — scales provided but the cache was never quantized to FP8, or the wrong cache tensor was passed.

Common situations: Enabling FP8 KV-cache quantization in config but the cache allocator still produced a bf16 pool; mixing quantized and non-quantized paths; stale cache from a previous non-quantized run.

Related errors

• unified_kv dtype mismatch: kv={unified_kv.dtype}, q={q.dtype
• Self attention has no KV cache scaling factor attribute!
• Self attention has no KV cache scaling factor attribute!
• Self attention has no KV cache scaling factor attribute!
• sparse_attn_v4_paged_decode expects fp16/bf16 q, got {q.dtyp

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/63693608a90551e8. Report an issue: GitHub.