sgl-project/sglang · error · ValueError

q can only be None when only_qv=True

Error message

q can only be None when only_qv=True

What it means

Symmetric guard on the query side: q may only be None when only_qv=True, where a dummy q is materialized from qv's shape and k_cache's head dim because the backend API still requires a q tensor even though the kernel ignores its values. A None q on the standard path is a caller bug.

Source

Thrown at python/sglang/kernels/aot/python/sgl_kernel/flash_attn.py:190

            k_dtype = q.dtype
            k_device = q.device
        elif k is not None:
            k_head_size = k.shape[-1]
            k_dtype = k.dtype
            k_device = k.device
        else:
            # Fallback: only_qv kernel ignores K values, so a tiny placeholder works.
            k_head_size = 64
            k_dtype = v_cache.dtype
            k_device = v_cache.device
        k_shape = (*v_cache.shape[:-1], k_head_size)
        # The kernel path for only_qv ignores K values, but backend API still requires k tensor.
        k_cache = torch.empty(k_shape, dtype=k_dtype, device=k_device)
    assert k_cache.stride(-1) == 1, "k_cache must have contiguous last dimension"

    if q is None:
        if not only_qv:
            raise ValueError("q can only be None when only_qv=True")
        if qv is None:
            raise ValueError(
                "q must be provided unless qv is provided with only_qv=True"
            )
        q_shape = (*qv.shape[:-1], k_cache.shape[-1])
        # The kernel path for only_qv ignores q values, but backend API still requires q tensor.
        q = torch.empty(q_shape, dtype=qv.dtype, device=qv.device)

    if softmax_scale is None:
        if only_qv:
            if qv is None:
                raise ValueError("only_qv=True requires qv to be provided")
            softmax_scale = (qv.shape[-1]) ** (-0.5)
        else:
            softmax_scale = (q.shape[-1] + (qv.shape[-1] if qv is not None else 0)) ** (
                -0.5
            )
    if cache_seqlens is not None and isinstance(cache_seqlens, int):

View on GitHub (pinned to 0132848349)

Solutions

1. Set only_qv=True and supply qv (and v_cache) for the query-value path.
2. Or pass a real q tensor with shape (..., nheads, head_dim).

Example fix

# before
flash_attn_with_kvcache(q=None, k_cache=kc, v_cache=vc)
# after
flash_attn_with_kvcache(q=None, only_qv=True, qv=qv, k_cache=kc, v_cache=vc)

if q is None:
    assert only_qv is True and qv is not None

def q_valid(q, only_qv: bool, qv) -> bool:
    return q is not None or (only_qv and qv is not None)

try:
    out = flash_attn_with_kvcache(...)
except ValueError as e:
    if "q can only be None" in str(e):
        out = flash_attn_with_kvcache(..., only_qv=True, qv=qv)

When it happens

Trigger: flash_attn_with_kvcache(q=None, ...) without only_qv=True; only_qv=True but qv also None (then the next guard fires); misordered keyword args so q receives None.

Common situations: QV-only attention models (e.g. MLA-like paths) integrated with the FA3 wrapper; refactors that stopped materializing q without setting the flag.

Related errors

• v_cache must be provided
• k_cache can only be None when only_qv=True
• q must be provided unless qv is provided with only_qv=True
• External ngram corpus max tokens must be positive.
• Unknown serve backend {name!r}. Available values: {available

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/f0e8c28e3c1dd042. Report an issue: GitHub.