sgl-project/sglang · error · NotImplementedError

FA4 path does not support rotary embedding.

Error message

FA4 path does not support rotary embedding.

What it means

Capability guard in the FA4 path: rotary embeddings must be applied before the attention call (pre-rotated q/k). FA4 kernels have no fused RoPE, so passing rotary arguments through this wrapper is rejected.

Solutions

1. Apply rotary embedding to q (and pre-rotated k in the cache) before calling FA4
2. Pass rotary_cos/rotary_sin/rotary_seqlens as None
3. Use FA2/FA3 backend if fused rope is needed

Example fix

# before
out = fa4.flash_attn_with_kvcache(q, k, v, kc, vc, cache_seqlens=s, rotary_cos=cos, rotary_sin=sin)
# after
q = apply_rotary(q, cos, sin)
out = fa4.flash_attn_with_kvcache(q, None, None, kc, vc, cache_seqlens=s)

assert rotary_cos is None and rotary_sin is None and rotary_seqlens is None, 'apply rope before FA4'

When it happens

Trigger: Calling flash_attn_with_kvcache(rotary_cos=..., rotary_sin=..., ...) or with rotary_seqlens on the FA4 backend.

Common situations: Attention wrappers that always pass rope tensors (as FA2/FA3 accept them) when the model uses rotary position embeddings.

Related errors

• FA4 path does not support rotary embedding.
• FA4 does not support updating KV cache in-place.
• FA4 does not support updating KV cache in-place.
• FA4 path does not support non-consecutive batch indices or…
• FA4 path does not support non-consecutive batch indices or…

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/bfac7911bb81073a. Report an issue: GitHub.

Appendix: source

Thrown at python/sglang/kernels/ops/attention/flash_attention_v4.py:266

    scheduler_metadata=None,
    num_splits: int = 0,
    pack_gqa: Optional[bool] = None,
    sm_margin: int = 0,
    sinks: Optional[torch.Tensor] = None,
    score_mod: Optional[Callable] = None,
    aux_tensors: Optional[list] = None,
    sfq: Optional[torch.Tensor] = None,
    sfk: Optional[torch.Tensor] = None,
    sfv: Optional[torch.Tensor] = None,
    rel_bias: Optional[torch.Tensor] = None,
    rel_bias_prep_cache: Optional[dict] = None,
    return_softmax_lse: bool = False,
    **_: object,
):
    if k is not None or v is not None:
        raise NotImplementedError("FA4 does not support updating KV cache in-place.")
    if rotary_cos is not None or rotary_sin is not None or rotary_seqlens is not None:
        raise NotImplementedError("FA4 path does not support rotary embedding.")
    if cache_batch_idx is not None or cache_leftpad is not None:
        raise NotImplementedError(
            "FA4 path does not support non-consecutive batch indices or left padding."
        )
    if isinstance(cache_seqlens, int):
        cache_seqlens = torch.full(
            (k_cache.shape[0],), cache_seqlens, dtype=torch.int32, device=k_cache.device
        )

    result = flash_attn_varlen_func(
        q=q,
        k=k_cache,
        v=v_cache,
        qv=qv,
        cu_seqlens_q=cu_seqlens_q,
        seqused_k=cache_seqlens,
        max_seqlen_q=max_seqlen_q,
        page_table=page_table,

View on GitHub (pinned to 0132848349)