{"record":{"id":"7b0e7170462abba5","repo":"sgl-project/sglang","slug":"fa4-path-does-not-support-non-consecutive-batch-in-7b0e71","errorCode":null,"errorMessage":"FA4 path does not support non-consecutive batch indices or left padding.","messagePattern":"FA4 path does not support non-consecutive batch indices or left padding\\.","errorType":"exception","errorClass":"NotImplementedError","httpStatus":null,"severity":"error","filePath":"python/sglang/kernels/ops/attention/flash_attention_v4_sm120.py","lineNumber":246,"sourceCode":"    score_mod: Optional[Callable] = None,\n    aux_tensors: Optional[list] = None,\n    sfq: Optional[torch.Tensor] = None,\n    sfk: Optional[torch.Tensor] = None,\n    sfv: Optional[torch.Tensor] = None,\n    rel_bias: Optional[torch.Tensor] = None,\n    rel_bias_prep_cache: Optional[dict] = None,\n    return_softmax_lse: bool = False,\n    out: Optional[torch.Tensor] = None,\n    max_seqlen_k: Optional[int] = None,\n    **_: object,\n):\n    _validate_out_contract(out)\n    if k is not None or v is not None:\n        raise NotImplementedError(\"FA4 does not support updating KV cache in-place.\")\n    if rotary_cos is not None or rotary_sin is not None or rotary_seqlens is not None:\n        raise NotImplementedError(\"FA4 path does not support rotary embedding.\")\n    if cache_batch_idx is not None or cache_leftpad is not None:\n        raise NotImplementedError(\n            \"FA4 path does not support non-consecutive batch indices or left padding.\"\n        )\n    if isinstance(cache_seqlens, int):\n        cache_seqlens = torch.full(\n            (k_cache.shape[0],), cache_seqlens, dtype=torch.int32, device=k_cache.device\n        )\n\n    forward_arch = get_forward_arch(q.device) if get_forward_arch is not None else None\n    if (\n        forward_arch is not None\n        and not return_softmax_lse\n        and softcap in (None, 0.0)\n        and all(\n            value is None\n            for value in (\n                qv,\n                score_mod,\n                aux_tensors,","sourceCodeStart":228,"sourceCodeEnd":264,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/kernels/ops/attention/flash_attention_v4_sm120.py#L228-L264","documentation":"The sm120 FA4 kvcache path requires consecutive batch slots: cache_batch_idx and cache_leftpad are not implemented and raise NotImplementedError when supplied.","triggerScenarios":"Calling flash_attn_with_kvcache(cache_batch_idx=..., ...) or cache_leftpad=... on the FA4 sm120 backend, e.g. with radix-cache scattered indices.","commonSituations":"Prefix/radix caching producing non-contiguous KV cache indices; left-padded batches for decoder-only models.","solutions":["Gather/reorder cache pages to consecutive slots and pass None for both args","Disable radix/prefix caching on the FA4 path","Use FA2/FA3 which support these arguments"],"exampleFix":"# before\nout = fa.flash_attn_with_kvcache(q, None, None, kc, vc, cache_seqlens=s, cache_batch_idx=idx)\n# after\nout = fa.flash_attn_with_kvcache(q, None, None, kc, vc, cache_seqlens=s)","handlingStrategy":"fallback","validationCode":"if cache_batch_idx is not None or cache_leftpad is not None:\\n    use_fa4_sm120 = False","typeGuard":null,"tryCatchPattern":"try:\\n    out = fa.flash_attn_with_kvcache(q, None, None, kc, vc, s, cache_batch_idx=idx)\\nexcept NotImplementedError:\\n    out = fa2.flash_attn_with_kvcache(q, None, None, kc, vc, s, cache_batch_idx=idx)","preventionTips":["Validate optional args against backend support matrix at init","Regression-test sparse cache indices against every enabled backend"],"tags":["flash-attention","fa4","sm120","batch-indices","unsupported-operation"],"backgroundTag":"unsupported-feature-in-backend","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}