sgl-project/sglang · error · NotImplementedError

FA4 path does not support non-consecutive batch indices or l

Error message

FA4 path does not support non-consecutive batch indices or left padding.

What it means

The sm120 FA4 kvcache path requires consecutive batch slots: cache_batch_idx and cache_leftpad are not implemented and raise NotImplementedError when supplied.

Source

Thrown at python/sglang/kernels/ops/attention/flash_attention_v4_sm120.py:246

    score_mod: Optional[Callable] = None,
    aux_tensors: Optional[list] = None,
    sfq: Optional[torch.Tensor] = None,
    sfk: Optional[torch.Tensor] = None,
    sfv: Optional[torch.Tensor] = None,
    rel_bias: Optional[torch.Tensor] = None,
    rel_bias_prep_cache: Optional[dict] = None,
    return_softmax_lse: bool = False,
    out: Optional[torch.Tensor] = None,
    max_seqlen_k: Optional[int] = None,
    **_: object,
):
    _validate_out_contract(out)
    if k is not None or v is not None:
        raise NotImplementedError("FA4 does not support updating KV cache in-place.")
    if rotary_cos is not None or rotary_sin is not None or rotary_seqlens is not None:
        raise NotImplementedError("FA4 path does not support rotary embedding.")
    if cache_batch_idx is not None or cache_leftpad is not None:
        raise NotImplementedError(
            "FA4 path does not support non-consecutive batch indices or left padding."
        )
    if isinstance(cache_seqlens, int):
        cache_seqlens = torch.full(
            (k_cache.shape[0],), cache_seqlens, dtype=torch.int32, device=k_cache.device
        )

    forward_arch = get_forward_arch(q.device) if get_forward_arch is not None else None
    if (
        forward_arch is not None
        and not return_softmax_lse
        and softcap in (None, 0.0)
        and all(
            value is None
            for value in (
                qv,
                score_mod,
                aux_tensors,

View on GitHub (pinned to 0132848349)

Solutions

1. Gather/reorder cache pages to consecutive slots and pass None for both args
2. Disable radix/prefix caching on the FA4 path
3. Use FA2/FA3 which support these arguments

Example fix

# before
out = fa.flash_attn_with_kvcache(q, None, None, kc, vc, cache_seqlens=s, cache_batch_idx=idx)
# after
out = fa.flash_attn_with_kvcache(q, None, None, kc, vc, cache_seqlens=s)

if cache_batch_idx is not None or cache_leftpad is not None:\n    use_fa4_sm120 = False

try:\n    out = fa.flash_attn_with_kvcache(q, None, None, kc, vc, s, cache_batch_idx=idx)\nexcept NotImplementedError:\n    out = fa2.flash_attn_with_kvcache(q, None, None, kc, vc, s, cache_batch_idx=idx)

When it happens

Trigger: Calling flash_attn_with_kvcache(cache_batch_idx=..., ...) or cache_leftpad=... on the FA4 sm120 backend, e.g. with radix-cache scattered indices.

Common situations: Prefix/radix caching producing non-contiguous KV cache indices; left-padded batches for decoder-only models.

Related errors

• FA4 path does not support non-consecutive batch indices or l
• FA4 does not support updating KV cache in-place.
• FA4 path does not support rotary embedding.
• FA4 does not support updating KV cache in-place.
• FA4 path does not support rotary embedding.

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/7b0e7170462abba5. Report an issue: GitHub.