{"record":{"id":"8fbdc71861afba69","repo":"sgl-project/sglang","slug":"k-cache-can-only-be-none-when-only-qv-true","errorCode":null,"errorMessage":"k_cache can only be None when only_qv=True","messagePattern":"k_cache can only be None when only_qv=True","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/kernels/aot/python/sgl_kernel/flash_attn.py","lineNumber":169,"sourceCode":"           Don't change this unless you know what you are doing.\n        return_softmax_lse: bool. Whether to return the logsumexp of the attention scores.\n        score_mod [optional]: A callable that takes the attention scores and applies a modification.\n        aux_tensors [optional]: Some score_mods will want to read from global aux_tensors. This is how we thread them through to the inner kernel.\n\n    Return:\n        out: (batch_size, seqlen, nheads, headdim).\n        softmax_lse [optional, if return_softmax_lse=True]: (batch_size, nheads, seqlen). The\n            logsumexp of each row of the matrix QK^T * scaling (e.g., log of the softmax\n            normalization factor).\n    \"\"\"\n\n    if v_cache is None:\n        raise ValueError(\"v_cache must be provided\")\n    assert v_cache.stride(-1) == 1, \"v_cache must have contiguous last dimension\"\n\n    if k_cache is None:\n        if not only_qv:\n            raise ValueError(\"k_cache can only be None when only_qv=True\")\n        if q is not None:\n            k_head_size = q.shape[-1]\n            k_dtype = q.dtype\n            k_device = q.device\n        elif k is not None:\n            k_head_size = k.shape[-1]\n            k_dtype = k.dtype\n            k_device = k.device\n        else:\n            # Fallback: only_qv kernel ignores K values, so a tiny placeholder works.\n            k_head_size = 64\n            k_dtype = v_cache.dtype\n            k_device = v_cache.device\n        k_shape = (*v_cache.shape[:-1], k_head_size)\n        # The kernel path for only_qv ignores K values, but backend API still requires k tensor.\n        k_cache = torch.empty(k_shape, dtype=k_dtype, device=k_device)\n    assert k_cache.stride(-1) == 1, \"k_cache must have contiguous last dimension\"\n","sourceCodeStart":151,"sourceCodeEnd":187,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/kernels/aot/python/sgl_kernel/flash_attn.py#L151-L187","documentation":"In flash_attn_with_kvcache, k_cache may only be None when only_qv=True — the query-value-only path where K is synthesized (its values are ignored by the kernel but the API needs a k tensor for shape/dtype/device inference). A None k_cache with only_qv False (default) is rejected.","triggerScenarios":"Calling flash_attn_with_kvcache(v_cache=..., k_cache=None) without only_qv=True; enabling only_qv but forgetting to set the flag; only_qv=True where neither q nor k is provided so k metadata can't be inferred (the code then derives head size/dtype from q or k).","commonSituations":"Linear-attention / query-value style models routed through the FA3 wrapper; partial migration where k_cache allocation was dropped but the flag wasn't added.","solutions":["If you genuinely don't have K, set only_qv=True and provide q (or k) plus qv and v_cache.","Otherwise pass a proper k_cache with stride(-1)==1 and matching shapes."],"exampleFix":"# before\nflash_attn_with_kvcache(q=q, v_cache=vc, k_cache=None)\n# after\nflash_attn_with_kvcache(q=q, v_cache=vc, only_qv=True, qv=qv)","handlingStrategy":"validation","validationCode":"if k_cache is None:\n    assert only_qv is True, \"k_cache=None requires only_qv=True\"","typeGuard":"def qv_path_valid(k_cache, only_qv: bool) -> bool:\n    return k_cache is not None or only_qv","tryCatchPattern":"try:\n    out = flash_attn_with_kvcache(...)\nexcept ValueError as e:\n    if \"k_cache can only be None\" in str(e):\n        out = flash_attn_with_kvcache(..., only_qv=True)","preventionTips":["Centralize only_qv flag decisions in one wrapper for QV-style models.","Assert cache presence invariants right after cache construction."],"tags":["sglang","flash-attention","kv-cache","invalid-argument"],"backgroundTag":"invalid-argument-combination","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}