sgl-project/sglang · error · RuntimeError

unified_kv dtype mismatch: kv={unified_kv.dtype}, q={q.dtype

Error message

unified_kv dtype mismatch: kv={unified_kv.dtype}, q={q.dtype}

What it means

In the non-quantized path (kv_scales=None), sparse_attn_v4_paged_decode requires unified_kv to have the same dtype as q (both fp16 or both bf16). Mixed dtypes would either break the Triton kernel or silently corrupt the attention math.

Source

Thrown at python/sglang/kernels/ops/attention/dsv4/unified_kv_kernels/paged_decode.py:684

            )
        if kv_scales.dtype != torch.float32:
            raise RuntimeError(f"kv_scales must be fp32, got {kv_scales.dtype}")
        D_check = unified_kv.shape[-1]
        if D_check % _FP8_GROUP_SIZE != 0:
            raise RuntimeError(
                f"D={D_check} must be divisible by GROUP_SIZE={_FP8_GROUP_SIZE}"
            )
        expected_g = D_check // _FP8_GROUP_SIZE
        if kv_scales.shape != (unified_kv.shape[0], expected_g):
            raise RuntimeError(
                f"kv_scales shape {tuple(kv_scales.shape)} does not match "
                f"expected ({unified_kv.shape[0]}, {expected_g})"
            )
        if kv_scales.stride(-1) != 1:
            kv_scales = kv_scales.contiguous()
    else:
        if unified_kv.dtype != q.dtype:
            raise RuntimeError(
                f"unified_kv dtype mismatch: kv={unified_kv.dtype}, q={q.dtype}"
            )

    T, H, D = q.shape
    out = torch.empty_like(q)

    if block_h is None:
        block_h = triton.next_power_of_2(min(H, 64))
    else:
        block_h = triton.next_power_of_2(block_h)
    block_h = max(block_h, 16)  # AMD MFMA min tile

    n_head_blocks = (H + block_h - 1) // block_h
    h_padded = n_head_blocks * block_h
    block_d = triton.next_power_of_2(D)

    if kv_splits is None:
        kv_splits = _kv_splits_heuristic(T, H, block_h)

View on GitHub (pinned to 0132848349)

Solutions

1. Cast unified_kv to q.dtype (or reallocate the cache pool with the model dtype)
2. Make the KV cache allocator derive its dtype from the model config dtype so they can never diverge
3. Verify server/model dtype settings are consistent end to end

Example fix

// before
out = sparse_attn_v4_paged_decode(q_bf16, kv_fp16)
// after
out = sparse_attn_v4_paged_decode(q_bf16, kv_fp16.to(torch.bfloat16))

if kv_scales is None:
    unified_kv = unified_kv.to(q.dtype)

def kv_dtype_ok(q, kv):
    return kv.dtype == q.dtype or (kv.dtype == torch.float8_e4m3fn)

When it happens

Trigger: Calling sparse_attn_v4_paged_decode with, e.g., q in bf16 and unified_kv in fp16, and no kv_scales.

Common situations: Model weights cast to bf16 while the KV cache pool was allocated fp16 (or vice versa); --dtype flag inconsistent with the cache allocator default; mixing checkpoints of different precision.

Related errors

• sparse_attn_v4_paged_decode expects fp16/bf16 q, got {q.dtyp
• kv_scales supplied but unified_kv is {unified_kv.dtype}, exp
• sparse_attn_v4_paged_prefill expects fp16/bf16 q, got {q.dty
• unified_kv dtype mismatch: kv={unified_kv.dtype}, q={q.dtype
• kv_scales must be fp32, got {kv_scales.dtype}

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/d66e149d3e38775a. Report an issue: GitHub.