sgl-project/sglang · error · ValueError

k_cache can only be None when only_qv=True

Error message

k_cache can only be None when only_qv=True

What it means

In flash_attn_with_kvcache, k_cache may only be None when only_qv=True — the query-value-only path where K is synthesized (its values are ignored by the kernel but the API needs a k tensor for shape/dtype/device inference). A None k_cache with only_qv False (default) is rejected.

Source

Thrown at python/sglang/kernels/aot/python/sgl_kernel/flash_attn.py:169

           Don't change this unless you know what you are doing.
        return_softmax_lse: bool. Whether to return the logsumexp of the attention scores.
        score_mod [optional]: A callable that takes the attention scores and applies a modification.
        aux_tensors [optional]: Some score_mods will want to read from global aux_tensors. This is how we thread them through to the inner kernel.

    Return:
        out: (batch_size, seqlen, nheads, headdim).
        softmax_lse [optional, if return_softmax_lse=True]: (batch_size, nheads, seqlen). The
            logsumexp of each row of the matrix QK^T * scaling (e.g., log of the softmax
            normalization factor).
    """

    if v_cache is None:
        raise ValueError("v_cache must be provided")
    assert v_cache.stride(-1) == 1, "v_cache must have contiguous last dimension"

    if k_cache is None:
        if not only_qv:
            raise ValueError("k_cache can only be None when only_qv=True")
        if q is not None:
            k_head_size = q.shape[-1]
            k_dtype = q.dtype
            k_device = q.device
        elif k is not None:
            k_head_size = k.shape[-1]
            k_dtype = k.dtype
            k_device = k.device
        else:
            # Fallback: only_qv kernel ignores K values, so a tiny placeholder works.
            k_head_size = 64
            k_dtype = v_cache.dtype
            k_device = v_cache.device
        k_shape = (*v_cache.shape[:-1], k_head_size)
        # The kernel path for only_qv ignores K values, but backend API still requires k tensor.
        k_cache = torch.empty(k_shape, dtype=k_dtype, device=k_device)
    assert k_cache.stride(-1) == 1, "k_cache must have contiguous last dimension"

View on GitHub (pinned to 0132848349)

Solutions

  1. If you genuinely don't have K, set only_qv=True and provide q (or k) plus qv and v_cache.
  2. Otherwise pass a proper k_cache with stride(-1)==1 and matching shapes.

Example fix

# before
flash_attn_with_kvcache(q=q, v_cache=vc, k_cache=None)
# after
flash_attn_with_kvcache(q=q, v_cache=vc, only_qv=True, qv=qv)
Defensive patterns

Strategy: validation

Validate before calling

if k_cache is None:
    assert only_qv is True, "k_cache=None requires only_qv=True"

Type guard

def qv_path_valid(k_cache, only_qv: bool) -> bool:
    return k_cache is not None or only_qv

Try / catch

try:
    out = flash_attn_with_kvcache(...)
except ValueError as e:
    if "k_cache can only be None" in str(e):
        out = flash_attn_with_kvcache(..., only_qv=True)

Prevention

When it happens

Trigger: Calling flash_attn_with_kvcache(v_cache=..., k_cache=None) without only_qv=True; enabling only_qv but forgetting to set the flag; only_qv=True where neither q nor k is provided so k metadata can't be inferred (the code then derives head size/dtype from q or k).

Common situations: Linear-attention / query-value style models routed through the FA3 wrapper; partial migration where k_cache allocation was dropped but the flag wasn't added.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/8fbdc71861afba69. Report an issue: GitHub.