sgl-project/sglang · error · NotImplementedError

FA4 path does not support non-consecutive batch indices or l

Error message

FA4 path does not support non-consecutive batch indices or left padding.

What it means

The FA4 backend assumes a contiguous batch: it rejects cache_batch_idx (gathered/non-consecutive batch indices) and cache_leftpad (left-padded sequences) with NotImplementedError.

Source

Thrown at python/sglang/kernels/ops/attention/flash_attention_v4.py:268

    pack_gqa: Optional[bool] = None,
    sm_margin: int = 0,
    sinks: Optional[torch.Tensor] = None,
    score_mod: Optional[Callable] = None,
    aux_tensors: Optional[list] = None,
    sfq: Optional[torch.Tensor] = None,
    sfk: Optional[torch.Tensor] = None,
    sfv: Optional[torch.Tensor] = None,
    rel_bias: Optional[torch.Tensor] = None,
    rel_bias_prep_cache: Optional[dict] = None,
    return_softmax_lse: bool = False,
    **_: object,
):
    if k is not None or v is not None:
        raise NotImplementedError("FA4 does not support updating KV cache in-place.")
    if rotary_cos is not None or rotary_sin is not None or rotary_seqlens is not None:
        raise NotImplementedError("FA4 path does not support rotary embedding.")
    if cache_batch_idx is not None or cache_leftpad is not None:
        raise NotImplementedError(
            "FA4 path does not support non-consecutive batch indices or left padding."
        )
    if isinstance(cache_seqlens, int):
        cache_seqlens = torch.full(
            (k_cache.shape[0],), cache_seqlens, dtype=torch.int32, device=k_cache.device
        )

    result = flash_attn_varlen_func(
        q=q,
        k=k_cache,
        v=v_cache,
        qv=qv,
        cu_seqlens_q=cu_seqlens_q,
        seqused_k=cache_seqlens,
        max_seqlen_q=max_seqlen_q,
        page_table=page_table,
        softmax_scale=softmax_scale,
        causal=causal,

View on GitHub (pinned to 0132848349)

Solutions

1. Pass cache_batch_idx=None and cache_leftpad=None; gather/reorder the cache pages so slots are consecutive
2. Disable the feature producing non-consecutive indices (e.g. radix cache) for the FA4 path
3. Fall back to FA2/FA3 backend which supports these arguments

Example fix

# before
out = fa4.flash_attn_with_kvcache(q, None, None, kc, vc, cache_seqlens=s, cache_batch_idx=idx)
# after
out = fa4.flash_attn_with_kvcache(q, None, None, kc, vc, cache_seqlens=s)  # gather cache first

if fa4_path and (cache_batch_idx is not None or cache_leftpad is not None):\n    use_fa4 = False  # fall back to FA2/FA3

try:\n    out = fa4.flash_attn_with_kvcache(q, None, None, kc, vc, s, cache_batch_idx=idx)\nexcept NotImplementedError:\n    out = fa2.flash_attn_with_kvcache(q, None, None, kc, vc, s, cache_batch_idx=idx)

When it happens

Trigger: Calling flash_attn_with_kvcache(cache_batch_idx=idx, ...) or cache_leftpad=pad on the FA4 path — common in radix-cache serving where requests map to non-contiguous cache slots.

Common situations: Serving with radix cache / prefix caching that produces scattered cache batch indices; padded batch layouts from tokenizers.

Related errors

• FA4 path does not support non-consecutive batch indices or l
• FA4 does not support updating KV cache in-place.
• FA4 path does not support rotary embedding.
• FA4 does not support updating KV cache in-place.
• FA4 path does not support rotary embedding.

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/d94ac199645ab229. Report an issue: GitHub.