sgl-project/sglang · error · ValueError

kv must be torch.float8_e4m3fn, got {kv.dtype}

Error message

kv must be torch.float8_e4m3fn, got {kv.dtype}

What it means

The key/value cache tensor kv must be torch.float8_e4m3fn for the Q8KV8 sparse prefill kernel, which reads KV directly as FP8 with e4m3 layout. Any other dtype (e.g. bf16, e5m2) is rejected up front to avoid garbage results or a kernel crash.

Source

Thrown at python/sglang/kernels/ops/attention/sparse_mla_q8kv8_prefill_sm90.py:333

    # accesses on q's CUDA device. Reject contract violations before launch.
    if not q.is_cuda:
        raise ValueError("q must be a CUDA tensor")
    if not kv.is_cuda:
        raise ValueError("kv must be a CUDA tensor")
    if not indices.is_cuda:
        raise ValueError("indices must be a CUDA tensor")

    if kv.device != device:
        raise ValueError(f"kv must be on q's device {device}, got {kv.device}")
    if indices.device != device:
        raise ValueError(
            f"indices must be on q's device {device}, got {indices.device}"
        )

    if q.dtype != torch.float8_e4m3fn:
        raise ValueError(f"q must be torch.float8_e4m3fn, got {q.dtype}")
    if kv.dtype != torch.float8_e4m3fn:
        raise ValueError(f"kv must be torch.float8_e4m3fn, got {kv.dtype}")

    if not q.is_contiguous():
        raise ValueError("q must be contiguous")
    if not kv.is_contiguous():
        raise ValueError("kv must be contiguous")
    if not indices.is_contiguous():
        raise ValueError("indices must be contiguous")

    if kv_d_qk != d_qk:
        raise ValueError(f"kv d_qk must match q d_qk={d_qk}, got {kv_d_qk}")

    # The CUDA implementation uses B_H=64 and launches h_q / B_H CTAs.
    # Reject unpadded TP-local head counts instead of launching zero CTAs and
    # returning uninitialized outputs, which can appear to callers as a hang or
    # a later collective failure.
    if h_q == 0 or h_q % 64 != 0:
        raise ValueError(
            "sparse_mla_q8kv8_prefill_fwd requires h_q padded to a positive "

View on GitHub (pinned to 0132848349)

Solutions

1. Ensure the KV cache tensor for this path is stored as torch.float8_e4m3fn (allocate/convert the cache accordingly)
2. Check your FP8 configuration selects e4m3, not e5m2
3. If you cannot use an FP8 cache, route prefill through the non-Q8 backend

Example fix

// before
out = sparse_mla_q8kv8_prefill_fwd(q_fp8, kv_bf16, indices)
// after
kv_fp8 = kv_bf16.to(torch.float8_e4m3fn)
out = sparse_mla_q8kv8_prefill_fwd(q_fp8, kv_fp8, indices)

assert kv.dtype == torch.float8_e4m3fn, f"kv dtype {kv.dtype}"

def is_q8kv8_cache(kv: torch.Tensor) -> bool:
    return kv.dtype == torch.float8_e4m3fn and kv.is_contiguous()

When it happens

Trigger: Passing a bf16 KV cache or an FP8 cache in torch.float8_e5m2 format to sparse_mla_q8kv8_prefill_fwd.

Common situations: The KV cache pool was allocated with the model dtype instead of the FP8 q8kv8 dtype; the e4m3 vs e5m2 FP8 variant was configured incorrectly (SGLANG FP8 format settings); mixing decode-path bf16 cache with the new sparse prefill kernel.

Related errors

• kv_scales must be fp32, got {kv_scales.dtype}
• q must be torch.float8_e4m3fn, got {q.dtype}
• kv_scales supplied but unified_kv is {unified_kv.dtype}, exp
• D={D_check} must be divisible by GROUP_SIZE={_FP8_GROUP_SIZE
• kv_scales shape {tuple(kv_scales.shape)} does not match expe

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/f1f5ada3bbfdb861. Report an issue: GitHub.