sgl-project/sglang · error · ValueError

kv must be contiguous

Error message

kv must be contiguous

What it means

The kv tensor must be contiguous because the SM90 kernel reads the FP8 KV cache via raw pointer arithmetic with no stride support. A non-contiguous kv view (slicing, padding, transpose) triggers this validation error.

Source

Thrown at python/sglang/kernels/ops/attention/sparse_mla_q8kv8_prefill_sm90.py:338

    if not indices.is_cuda:
        raise ValueError("indices must be a CUDA tensor")

    if kv.device != device:
        raise ValueError(f"kv must be on q's device {device}, got {kv.device}")
    if indices.device != device:
        raise ValueError(
            f"indices must be on q's device {device}, got {indices.device}"
        )

    if q.dtype != torch.float8_e4m3fn:
        raise ValueError(f"q must be torch.float8_e4m3fn, got {q.dtype}")
    if kv.dtype != torch.float8_e4m3fn:
        raise ValueError(f"kv must be torch.float8_e4m3fn, got {kv.dtype}")

    if not q.is_contiguous():
        raise ValueError("q must be contiguous")
    if not kv.is_contiguous():
        raise ValueError("kv must be contiguous")
    if not indices.is_contiguous():
        raise ValueError("indices must be contiguous")

    if kv_d_qk != d_qk:
        raise ValueError(f"kv d_qk must match q d_qk={d_qk}, got {kv_d_qk}")

    # The CUDA implementation uses B_H=64 and launches h_q / B_H CTAs.
    # Reject unpadded TP-local head counts instead of launching zero CTAs and
    # returning uninitialized outputs, which can appear to callers as a hang or
    # a later collective failure.
    if h_q == 0 or h_q % 64 != 0:
        raise ValueError(
            "sparse_mla_q8kv8_prefill_fwd requires h_q padded to a positive "
            f"multiple of 64, got {h_q}"
        )

    if h_kv != 1:
        raise ValueError(f"sparse_mla_q8kv8_prefill_fwd requires h_kv=1, got {h_kv}")

View on GitHub (pinned to 0132848349)

Solutions

1. Materialize a contiguous copy: kv = kv.contiguous() (or ensure the pool slice is contiguous)
2. Verify the cache layout matches what the sparse prefill backend expects (HND with unit inner stride)
3. Reorder/repad the cache allocation so the region passed to the kernel is dense

Example fix

// before
out = sparse_mla_q8kv8_prefill_fwd(q, kv_view, indices)
// after
kv = kv_view.contiguous()
out = sparse_mla_q8kv8_prefill_fwd(q, kv, indices)

if not kv.is_contiguous(): kv = kv.contiguous()

def kv_ready(kv: torch.Tensor) -> bool:
    return kv.is_contiguous() and kv.dtype == torch.float8_e4m3fn

When it happens

Trigger: Passing kv as a strided view, e.g. kv_cache[batch_start:batch_end] with non-unit stride in the last dim, or a cache laid out NHoHd that was permuted without materializing.

Common situations: Paged/HiCache KV layouts where a per-request view into a larger pool is non-contiguous; slicing d_qk/d_v planes out of a combined cache buffer; version changes in cache layout.

Related errors

• The layout of k is not supported
• The layout of v is not supported
• Type mismatch: {self.q_dtype} != {self.v_dtype}
• MXFP8 fused prologue requires contiguous interleaved SFK/SFV
• q must be contiguous

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/721271c3acba8b55. Report an issue: GitHub.