{"record":{"id":"d1d80112569429d6","repo":"sgl-project/sglang","slug":"fa4-does-not-support-updating-kv-cache-in-place","errorCode":null,"errorMessage":"FA4 does not support updating KV cache in-place.","messagePattern":"FA4 does not support updating KV cache in-place\\.","errorType":"validation","errorClass":"NotImplementedError","httpStatus":null,"severity":"error","filePath":"python/sglang/kernels/ops/attention/flash_attention_v4.py","lineNumber":264,"sourceCode":"    softcap: float = 0.0,\n    rotary_interleaved: bool = True,\n    scheduler_metadata=None,\n    num_splits: int = 0,\n    pack_gqa: Optional[bool] = None,\n    sm_margin: int = 0,\n    sinks: Optional[torch.Tensor] = None,\n    score_mod: Optional[Callable] = None,\n    aux_tensors: Optional[list] = None,\n    sfq: Optional[torch.Tensor] = None,\n    sfk: Optional[torch.Tensor] = None,\n    sfv: Optional[torch.Tensor] = None,\n    rel_bias: Optional[torch.Tensor] = None,\n    rel_bias_prep_cache: Optional[dict] = None,\n    return_softmax_lse: bool = False,\n    **_: object,\n):\n    if k is not None or v is not None:\n        raise NotImplementedError(\"FA4 does not support updating KV cache in-place.\")\n    if rotary_cos is not None or rotary_sin is not None or rotary_seqlens is not None:\n        raise NotImplementedError(\"FA4 path does not support rotary embedding.\")\n    if cache_batch_idx is not None or cache_leftpad is not None:\n        raise NotImplementedError(\n            \"FA4 path does not support non-consecutive batch indices or left padding.\"\n        )\n    if isinstance(cache_seqlens, int):\n        cache_seqlens = torch.full(\n            (k_cache.shape[0],), cache_seqlens, dtype=torch.int32, device=k_cache.device\n        )\n\n    result = flash_attn_varlen_func(\n        q=q,\n        k=k_cache,\n        v=v_cache,\n        qv=qv,\n        cu_seqlens_q=cu_seqlens_q,\n        seqused_k=cache_seqlens,","sourceCodeStart":246,"sourceCodeEnd":282,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/kernels/ops/attention/flash_attention_v4.py#L246-L282","documentation":"The FA4 kvcache API expects the KV cache to be pre-filled; it does not accept new k/v tensors to append into the cache in-place (unlike FA2/FA3 wrappers). Passing k or v raises NotImplementedError.","triggerScenarios":"Calling flash_attn_with_kvcache(k=..., v=..., k_cache=..., ...) on the FA4 path — i.e. supplying new keys/values alongside the paged cache.","commonSituations":"Code ported from FA2/FA3 call sites that pass k/v for cache update during prefill/decode; generic attention backends that always supply k/v.","solutions":["Pre-write k/v into the paged KV cache yourself (scatter into k_cache/v_cache) and call with k=None, v=None","Use cache_seqlens to describe valid lengths instead of appending","Switch to FA2/FA3 backend if in-place append semantics are required"],"exampleFix":"# before\nout = flash_attn_with_kvcache(q, k, v, k_cache, v_cache, cache_seqlens=seqlens)\n# after (FA4): write k,v into cache first, then\nout = flash_attn_with_kvcache(q, None, None, k_cache, v_cache, cache_seqlens=seqlens)","handlingStrategy":"validation","validationCode":"assert k is None and v is None, 'FA4 requires pre-filled cache; write k/v into k_cache first'","typeGuard":null,"tryCatchPattern":"try:\\n    out = fa4.flash_attn_with_kvcache(q, k, v, kc, vc, s)\\nexcept NotImplementedError:\\n    scatter_kv_into_cache(kc, vc, k, v)\\n    out = fa4.flash_attn_with_kvcache(q, None, None, kc, vc, s)","preventionTips":["Never pass k/v on FA4 call sites","Centralize cache-write logic before backend dispatch"],"tags":["flash-attention","fa4","kv-cache","unsupported-operation"],"backgroundTag":"unsupported-kv-cache-update","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}