sgl-project/sglang · error · RuntimeError

sparse_attn_v4_paged_decode expects fp16/bf16 q, got {q.dtyp

Error message

sparse_attn_v4_paged_decode expects fp16/bf16 q, got {q.dtype}

What it means

The Triton sparse DeepSeek-V4 paged decode attention kernel only accepts query tensors in torch.float16 or torch.bfloat16. Any other dtype (fp32, fp8, etc.) is rejected up front because the kernel's Triton code is specialized for 16-bit inputs. This mirrors the CPU/CUDA guard immediately above it.

Source

Thrown at python/sglang/kernels/ops/attention/dsv4/unified_kv_kernels/paged_decode.py:656

    block_h: int | None = None,
    kv_splits: int | None = None,
    block_k: int | None = None,
) -> torch.Tensor:
    """V4 sparse decode Triton implementation: split-K with FUSED fast path,
    exp2 softmax, CG-safe heuristic. ``block_h`` and ``kv_splits`` are
    escape hatches for benchmarks; production callers pass neither.

    When ``kv_scales`` is provided, ``unified_kv`` must be e4m3fnuz and
    ``kv_scales`` must be ``[total_pages, D // GROUP_SIZE]`` fp32 — 1xGROUP_SIZE
    block-scale quantization. Dequant happens in-kernel; the dot still runs
    in q.dtype.
    """
    if not q.is_cuda:
        raise RuntimeError(
            "Triton sparse_attn_v4_paged_decode requires CUDA/HIP tensors"
        )
    if q.dtype not in (torch.bfloat16, torch.float16):
        raise RuntimeError(
            f"sparse_attn_v4_paged_decode expects fp16/bf16 q, got {q.dtype}"
        )

    quant_kv = kv_scales is not None
    if quant_kv:
        if unified_kv.dtype != _FP8_DTYPE:
            raise RuntimeError(
                f"kv_scales supplied but unified_kv is {unified_kv.dtype}, "
                f"expected {_FP8_DTYPE}"
            )
        if kv_scales.dtype != torch.float32:
            raise RuntimeError(f"kv_scales must be fp32, got {kv_scales.dtype}")
        D_check = unified_kv.shape[-1]
        if D_check % _FP8_GROUP_SIZE != 0:
            raise RuntimeError(
                f"D={D_check} must be divisible by GROUP_SIZE={_FP8_GROUP_SIZE}"
            )
        expected_g = D_check // _FP8_GROUP_SIZE

View on GitHub (pinned to 0132848349)

Solutions

Cast q to bf16/fp16 before calling: q = q.to(torch.bfloat16)
Check the model's dtype configuration (server --dtype) matches a supported 16-bit dtype
If you control the caller, add a dtype assertion early so the failure points at the producer, not the kernel

Example fix

// before
out = sparse_attn_v4_paged_decode(q, ...)  # q is float32
// after
out = sparse_attn_v4_paged_decode(q.to(torch.bfloat16), ...)

Defensive patterns

Strategy: type-guard

Validate before calling

assert q.is_cuda and q.dtype in (torch.float16, torch.bfloat16), f"q must be cuda fp16/bf16, got {q.device} {q.dtype}"

Type guard

def is_q_valid(q: torch.Tensor) -> bool:
    return q.is_cuda and q.dtype in (torch.float16, torch.bfloat16)

Prevention

Standardize one compute dtype (bf16) for the whole model and assert it at entry points
Add dtype/device asserts in test fixtures, not just in the kernel wrapper

When it happens

Trigger: Calling sparse_attn_v4_paged_decode with a q tensor whose dtype is not fp16/bf16, e.g. a model or test that produced float32 queries, or a quantized path that passes fp8 q directly.

Common situations: Unit tests building fp32 fixtures; a model variant that leaves q in fp32; accidentally passing a dequantized/upscore tensor; running with a dtype override like --dtype float32.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/dfccdd39b642f807. Report an issue: GitHub.