sgl-project/sglang · error · RuntimeError

unified_kv dtype mismatch: kv={unified_kv.dtype}, q={q.dtype

Error message

unified_kv dtype mismatch: kv={unified_kv.dtype}, q={q.dtype}

What it means

In sparse_attn_v4_paged_prefill, the paged unified_kv cache must share q's dtype. The kernel does no implicit conversion, so a dtype mismatch aborts the call.

Source

Thrown at python/sglang/kernels/ops/attention/dsv4/unified_kv_kernels/paged_prefill.py:237

    unified_kv: torch.Tensor,
    kv_indices_prefix: torch.Tensor,
    kv_indptr_prefix: torch.Tensor,
    kv: torch.Tensor,
    kv_indices_extend: torch.Tensor,
    kv_indptr_extend: torch.Tensor,
    attn_sink: torch.Tensor,
    softmax_scale: float,
) -> torch.Tensor:
    if not q.is_cuda:
        raise RuntimeError(
            "Triton sparse_attn_v4_paged_prefill requires CUDA/HIP tensors"
        )
    if q.dtype not in (torch.bfloat16, torch.float16):
        raise RuntimeError(
            f"sparse_attn_v4_paged_prefill expects fp16/bf16 q, got {q.dtype}"
        )
    if unified_kv.dtype != q.dtype:
        raise RuntimeError(
            f"unified_kv dtype mismatch: kv={unified_kv.dtype}, q={q.dtype}"
        )
    if kv.dtype != q.dtype:
        raise RuntimeError(f"kv dtype mismatch: kv={kv.dtype}, q={q.dtype}")
    if unified_kv.size(-1) != kv.size(-1):
        raise RuntimeError(
            f"head_dim mismatch: unified_kv={unified_kv.size(-1)}, kv={kv.size(-1)}"
        )

    T, H, D = q.shape
    out = torch.empty_like(q)
    kv_indices_prefix = kv_indices_prefix.to(torch.int32).contiguous()
    kv_indptr_prefix = kv_indptr_prefix.to(torch.int32).contiguous()
    kv_indices_extend = kv_indices_extend.to(torch.int32).contiguous()
    kv_indptr_extend = kv_indptr_extend.to(torch.int32).contiguous()

    block_h = 16  # AMD MFMA min tile
    block_d = triton.next_power_of_2(D)

View on GitHub (pinned to 0132848349)

Solutions

Cast/reallocate unified_kv to match q.dtype
Derive the cache dtype from the model config dtype so they cannot diverge
Double-check server --dtype vs cache allocator defaults

Example fix

// before
out = sparse_attn_v4_paged_prefill(q_bf16, kv_fp16, ...)
// after
out = sparse_attn_v4_paged_prefill(q_bf16, kv_fp16.to(torch.bfloat16), ...)

Defensive patterns

Strategy: validation

Validate before calling

unified_kv = unified_kv.to(q.dtype)

Prevention

Derive cache dtype from model config dtype in one place
Add a preflight check that q, unified_kv, kv all share dtype

When it happens

Trigger: Calling sparse_attn_v4_paged_prefill with q in one 16-bit dtype and unified_kv in the other (fp16 vs bf16) or in fp32.

Common situations: KV cache pool allocated with a different default dtype than the model weights; mixed-precision experiments; checkpoint precision differing from server --dtype.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/ca7c34e0ea160c45. Report an issue: GitHub.