sgl-project/sglang · error · RuntimeError

sparse_attn_v4_paged_prefill expects fp16/bf16 q, got {q.dty

Error message

sparse_attn_v4_paged_prefill expects fp16/bf16 q, got {q.dtype}

What it means

The Triton sparse_attn_v4_paged_prefill kernel only supports fp16/bf16 queries; any other q dtype is rejected before kernel launch because the Triton code is specialized for 16-bit math.

Source

Thrown at python/sglang/kernels/ops/attention/dsv4/unified_kv_kernels/paged_prefill.py:233


def _sparse_attn_v4_paged_prefill_triton(
    q: torch.Tensor,
    unified_kv: torch.Tensor,
    kv_indices_prefix: torch.Tensor,
    kv_indptr_prefix: torch.Tensor,
    kv: torch.Tensor,
    kv_indices_extend: torch.Tensor,
    kv_indptr_extend: torch.Tensor,
    attn_sink: torch.Tensor,
    softmax_scale: float,
) -> torch.Tensor:
    if not q.is_cuda:
        raise RuntimeError(
            "Triton sparse_attn_v4_paged_prefill requires CUDA/HIP tensors"
        )
    if q.dtype not in (torch.bfloat16, torch.float16):
        raise RuntimeError(
            f"sparse_attn_v4_paged_prefill expects fp16/bf16 q, got {q.dtype}"
        )
    if unified_kv.dtype != q.dtype:
        raise RuntimeError(
            f"unified_kv dtype mismatch: kv={unified_kv.dtype}, q={q.dtype}"
        )
    if kv.dtype != q.dtype:
        raise RuntimeError(f"kv dtype mismatch: kv={kv.dtype}, q={q.dtype}")
    if unified_kv.size(-1) != kv.size(-1):
        raise RuntimeError(
            f"head_dim mismatch: unified_kv={unified_kv.size(-1)}, kv={kv.size(-1)}"
        )

    T, H, D = q.shape
    out = torch.empty_like(q)
    kv_indices_prefix = kv_indices_prefix.to(torch.int32).contiguous()
    kv_indptr_prefix = kv_indptr_prefix.to(torch.int32).contiguous()
    kv_indices_extend = kv_indices_extend.to(torch.int32).contiguous()

View on GitHub (pinned to 0132848349)

Solutions

Cast q to torch.bfloat16 or torch.float16 before the call
Align the model/dtype configuration with a supported 16-bit dtype
Add an early dtype assert at the call site to catch the producer

Example fix

// before
out = sparse_attn_v4_paged_prefill(q_fp32, ...)
// after
out = sparse_attn_v4_paged_prefill(q_fp32.to(torch.bfloat16), ...)

Defensive patterns

Strategy: validation

Validate before calling

if q.dtype not in (torch.float16, torch.bfloat16):
    q = q.to(torch.bfloat16)

Type guard

def q_dtype_ok(q: torch.Tensor) -> bool:
    return q.dtype in (torch.float16, torch.bfloat16)

Prevention

Keep a shared dtype guard helper for all dsv4 attention call sites
Avoid --dtype float32 for models routed to these Triton kernels

When it happens

Trigger: Calling sparse_attn_v4_paged_prefill with q in float32 (or fp8) dtype.

Common situations: Prefill fixtures in fp32 in tests; a model configuration leaving hidden states in fp32; a --dtype float32 run; passing a logits-scale or normalized tensor of the wrong dtype.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/31070eef9df6dbb9. Report an issue: GitHub.