{"record":{"id":"d94ac199645ab229","repo":"sgl-project/sglang","slug":"fa4-path-does-not-support-non-consecutive-batch-in","errorCode":null,"errorMessage":"FA4 path does not support non-consecutive batch indices or left padding.","messagePattern":"FA4 path does not support non-consecutive batch indices or left padding\\.","errorType":"exception","errorClass":"NotImplementedError","httpStatus":null,"severity":"error","filePath":"python/sglang/kernels/ops/attention/flash_attention_v4.py","lineNumber":268,"sourceCode":"    pack_gqa: Optional[bool] = None,\n    sm_margin: int = 0,\n    sinks: Optional[torch.Tensor] = None,\n    score_mod: Optional[Callable] = None,\n    aux_tensors: Optional[list] = None,\n    sfq: Optional[torch.Tensor] = None,\n    sfk: Optional[torch.Tensor] = None,\n    sfv: Optional[torch.Tensor] = None,\n    rel_bias: Optional[torch.Tensor] = None,\n    rel_bias_prep_cache: Optional[dict] = None,\n    return_softmax_lse: bool = False,\n    **_: object,\n):\n    if k is not None or v is not None:\n        raise NotImplementedError(\"FA4 does not support updating KV cache in-place.\")\n    if rotary_cos is not None or rotary_sin is not None or rotary_seqlens is not None:\n        raise NotImplementedError(\"FA4 path does not support rotary embedding.\")\n    if cache_batch_idx is not None or cache_leftpad is not None:\n        raise NotImplementedError(\n            \"FA4 path does not support non-consecutive batch indices or left padding.\"\n        )\n    if isinstance(cache_seqlens, int):\n        cache_seqlens = torch.full(\n            (k_cache.shape[0],), cache_seqlens, dtype=torch.int32, device=k_cache.device\n        )\n\n    result = flash_attn_varlen_func(\n        q=q,\n        k=k_cache,\n        v=v_cache,\n        qv=qv,\n        cu_seqlens_q=cu_seqlens_q,\n        seqused_k=cache_seqlens,\n        max_seqlen_q=max_seqlen_q,\n        page_table=page_table,\n        softmax_scale=softmax_scale,\n        causal=causal,","sourceCodeStart":250,"sourceCodeEnd":286,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/kernels/ops/attention/flash_attention_v4.py#L250-L286","documentation":"The FA4 backend assumes a contiguous batch: it rejects cache_batch_idx (gathered/non-consecutive batch indices) and cache_leftpad (left-padded sequences) with NotImplementedError.","triggerScenarios":"Calling flash_attn_with_kvcache(cache_batch_idx=idx, ...) or cache_leftpad=pad on the FA4 path — common in radix-cache serving where requests map to non-contiguous cache slots.","commonSituations":"Serving with radix cache / prefix caching that produces scattered cache batch indices; padded batch layouts from tokenizers.","solutions":["Pass cache_batch_idx=None and cache_leftpad=None; gather/reorder the cache pages so slots are consecutive","Disable the feature producing non-consecutive indices (e.g. radix cache) for the FA4 path","Fall back to FA2/FA3 backend which supports these arguments"],"exampleFix":"# before\nout = fa4.flash_attn_with_kvcache(q, None, None, kc, vc, cache_seqlens=s, cache_batch_idx=idx)\n# after\nout = fa4.flash_attn_with_kvcache(q, None, None, kc, vc, cache_seqlens=s)  # gather cache first","handlingStrategy":"fallback","validationCode":"if fa4_path and (cache_batch_idx is not None or cache_leftpad is not None):\\n    use_fa4 = False  # fall back to FA2/FA3","typeGuard":null,"tryCatchPattern":"try:\\n    out = fa4.flash_attn_with_kvcache(q, None, None, kc, vc, s, cache_batch_idx=idx)\\nexcept NotImplementedError:\\n    out = fa2.flash_attn_with_kvcache(q, None, None, kc, vc, s, cache_batch_idx=idx)","preventionTips":["Detect radix-cache scatter and route away from FA4","Test backend selection with prefix caching enabled"],"tags":["flash-attention","fa4","batch-indices","left-padding","unsupported-operation"],"backgroundTag":"unsupported-feature-in-backend","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}