sgl-project/sglang · error · ValueError

q must be torch.float8_e4m3fn, got {q.dtype}

Error message

q must be torch.float8_e4m3fn, got {q.dtype}

What it means

The sparse Q8KV8 prefill kernel (SM90) quantizes and computes in FP8, so the query tensor q must be torch.float8_e4m3fn. Passing bf16/fp16/fp32 queries is rejected because the kernel has no dequant/quant path for q.

Source

Thrown at python/sglang/kernels/ops/attention/sparse_mla_q8kv8_prefill_sm90.py:331

    # entry.cuh interprets q/kv as contiguous FP8 buffers and launches all
    # accesses on q's CUDA device. Reject contract violations before launch.
    if not q.is_cuda:
        raise ValueError("q must be a CUDA tensor")
    if not kv.is_cuda:
        raise ValueError("kv must be a CUDA tensor")
    if not indices.is_cuda:
        raise ValueError("indices must be a CUDA tensor")

    if kv.device != device:
        raise ValueError(f"kv must be on q's device {device}, got {kv.device}")
    if indices.device != device:
        raise ValueError(
            f"indices must be on q's device {device}, got {indices.device}"
        )

    if q.dtype != torch.float8_e4m3fn:
        raise ValueError(f"q must be torch.float8_e4m3fn, got {q.dtype}")
    if kv.dtype != torch.float8_e4m3fn:
        raise ValueError(f"kv must be torch.float8_e4m3fn, got {kv.dtype}")

    if not q.is_contiguous():
        raise ValueError("q must be contiguous")
    if not kv.is_contiguous():
        raise ValueError("kv must be contiguous")
    if not indices.is_contiguous():
        raise ValueError("indices must be contiguous")

    if kv_d_qk != d_qk:
        raise ValueError(f"kv d_qk must match q d_qk={d_qk}, got {kv_d_qk}")

    # The CUDA implementation uses B_H=64 and launches h_q / B_H CTAs.
    # Reject unpadded TP-local head counts instead of launching zero CTAs and
    # returning uninitialized outputs, which can appear to callers as a hang or
    # a later collective failure.
    if h_q == 0 or h_q % 64 != 0:

View on GitHub (pinned to 0132848349)

Solutions

1. Quantize q to float8_e4m3fn with the same scale scheme used for kv before calling (e.g. q.to(torch.float8_e4m3fn) or the framework's fp8 quantize helper)
2. Ensure your backend's q8kv8 sparse path actually enables FP8 quantization of q, not just kv
3. If you only have bf16 tensors, use the non-Q8 non-sparse prefill path instead of this kernel

Example fix

// before
q_fp8 = q  # still bfloat16
out = sparse_mla_q8kv8_prefill_fwd(q_fp8, kv_fp8, indices)
// after
q_fp8 = q.to(torch.float8_e4m3fn)
out = sparse_mla_q8kv8_prefill_fwd(q_fp8, kv_fp8, indices)

if q.dtype != torch.float8_e4m3fn: q = q.to(torch.float8_e4m3fn)

def is_fp8_e4m3(t: torch.Tensor) -> bool:
    return t.dtype == torch.float8_e4m3fn

When it happens

Trigger: Calling sparse_mla_q8kv8_prefill_fwd with a q tensor in torch.bfloat16 (e.g. reusing the decode-path q tensor that was never quantized to FP8).

Common situations: Wiring a new model/attention backend where q is produced in the default model dtype (bf16) and the FP8 quantization step (per-token or static scale cast to float8_e4m3fn) was skipped; version changes that moved quantization into a separate op.

Related errors

• kv must be torch.float8_e4m3fn, got {kv.dtype}
• kv_scales must be fp32, got {kv_scales.dtype}
• indices must be on q's device {device}, got {indices.device}
• indices must be int32, got {indices.dtype}
• attn_sink must be float32 with shape ({h_q},), got {tuple(at

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/78f02a647385ab76. Report an issue: GitHub.