sgl-project/sglang · error · NotImplementedError

FA4 does not support updating KV cache in-place.

Error message

FA4 does not support updating KV cache in-place.

What it means

The FA4 kvcache API expects the KV cache to be pre-filled; it does not accept new k/v tensors to append into the cache in-place (unlike FA2/FA3 wrappers). Passing k or v raises NotImplementedError.

Source

Thrown at python/sglang/kernels/ops/attention/flash_attention_v4.py:264

    softcap: float = 0.0,
    rotary_interleaved: bool = True,
    scheduler_metadata=None,
    num_splits: int = 0,
    pack_gqa: Optional[bool] = None,
    sm_margin: int = 0,
    sinks: Optional[torch.Tensor] = None,
    score_mod: Optional[Callable] = None,
    aux_tensors: Optional[list] = None,
    sfq: Optional[torch.Tensor] = None,
    sfk: Optional[torch.Tensor] = None,
    sfv: Optional[torch.Tensor] = None,
    rel_bias: Optional[torch.Tensor] = None,
    rel_bias_prep_cache: Optional[dict] = None,
    return_softmax_lse: bool = False,
    **_: object,
):
    if k is not None or v is not None:
        raise NotImplementedError("FA4 does not support updating KV cache in-place.")
    if rotary_cos is not None or rotary_sin is not None or rotary_seqlens is not None:
        raise NotImplementedError("FA4 path does not support rotary embedding.")
    if cache_batch_idx is not None or cache_leftpad is not None:
        raise NotImplementedError(
            "FA4 path does not support non-consecutive batch indices or left padding."
        )
    if isinstance(cache_seqlens, int):
        cache_seqlens = torch.full(
            (k_cache.shape[0],), cache_seqlens, dtype=torch.int32, device=k_cache.device
        )

    result = flash_attn_varlen_func(
        q=q,
        k=k_cache,
        v=v_cache,
        qv=qv,
        cu_seqlens_q=cu_seqlens_q,
        seqused_k=cache_seqlens,

View on GitHub (pinned to 0132848349)

Solutions

1. Pre-write k/v into the paged KV cache yourself (scatter into k_cache/v_cache) and call with k=None, v=None
2. Use cache_seqlens to describe valid lengths instead of appending
3. Switch to FA2/FA3 backend if in-place append semantics are required

Example fix

# before
out = flash_attn_with_kvcache(q, k, v, k_cache, v_cache, cache_seqlens=seqlens)
# after (FA4): write k,v into cache first, then
out = flash_attn_with_kvcache(q, None, None, k_cache, v_cache, cache_seqlens=seqlens)

assert k is None and v is None, 'FA4 requires pre-filled cache; write k/v into k_cache first'

try:\n    out = fa4.flash_attn_with_kvcache(q, k, v, kc, vc, s)\nexcept NotImplementedError:\n    scatter_kv_into_cache(kc, vc, k, v)\n    out = fa4.flash_attn_with_kvcache(q, None, None, kc, vc, s)

When it happens

Trigger: Calling flash_attn_with_kvcache(k=..., v=..., k_cache=..., ...) on the FA4 path — i.e. supplying new keys/values alongside the paged cache.

Common situations: Code ported from FA2/FA3 call sites that pass k/v for cache update during prefill/decode; generic attention backends that always supply k/v.

Related errors

• FA4 does not support updating KV cache in-place.
• FA4 path does not support rotary embedding.
• FA4 path does not support non-consecutive batch indices or l
• FA4 path does not support rotary embedding.
• FA4 path does not support non-consecutive batch indices or l

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/d1d80112569429d6. Report an issue: GitHub.