sgl-project/sglang · error · NotImplementedError

FA4 path does not support rotary embedding.

Error message

FA4 path does not support rotary embedding.

What it means

The FA4 path does not implement rotary embedding fused into the attention kernel. Passing rotary_cos/rotary_sin/rotary_seqlens raises NotImplementedError.

Source

Thrown at python/sglang/kernels/ops/attention/flash_attention_v4.py:266

    scheduler_metadata=None,
    num_splits: int = 0,
    pack_gqa: Optional[bool] = None,
    sm_margin: int = 0,
    sinks: Optional[torch.Tensor] = None,
    score_mod: Optional[Callable] = None,
    aux_tensors: Optional[list] = None,
    sfq: Optional[torch.Tensor] = None,
    sfk: Optional[torch.Tensor] = None,
    sfv: Optional[torch.Tensor] = None,
    rel_bias: Optional[torch.Tensor] = None,
    rel_bias_prep_cache: Optional[dict] = None,
    return_softmax_lse: bool = False,
    **_: object,
):
    if k is not None or v is not None:
        raise NotImplementedError("FA4 does not support updating KV cache in-place.")
    if rotary_cos is not None or rotary_sin is not None or rotary_seqlens is not None:
        raise NotImplementedError("FA4 path does not support rotary embedding.")
    if cache_batch_idx is not None or cache_leftpad is not None:
        raise NotImplementedError(
            "FA4 path does not support non-consecutive batch indices or left padding."
        )
    if isinstance(cache_seqlens, int):
        cache_seqlens = torch.full(
            (k_cache.shape[0],), cache_seqlens, dtype=torch.int32, device=k_cache.device
        )

    result = flash_attn_varlen_func(
        q=q,
        k=k_cache,
        v=v_cache,
        qv=qv,
        cu_seqlens_q=cu_seqlens_q,
        seqused_k=cache_seqlens,
        max_seqlen_q=max_seqlen_q,
        page_table=page_table,

View on GitHub (pinned to 0132848349)

Solutions

1. Apply rotary embedding to q (and pre-rotated k in the cache) before calling FA4
2. Pass rotary_cos/rotary_sin/rotary_seqlens as None
3. Use FA2/FA3 backend if fused rope is needed

Example fix

# before
out = fa4.flash_attn_with_kvcache(q, k, v, kc, vc, cache_seqlens=s, rotary_cos=cos, rotary_sin=sin)
# after
q = apply_rotary(q, cos, sin)
out = fa4.flash_attn_with_kvcache(q, None, None, kc, vc, cache_seqlens=s)

assert rotary_cos is None and rotary_sin is None and rotary_seqlens is None, 'apply rope before FA4'

When it happens

Trigger: Calling flash_attn_with_kvcache(rotary_cos=..., rotary_sin=..., ...) or with rotary_seqlens on the FA4 backend.

Common situations: Attention wrappers that always pass rope tensors (as FA2/FA3 accept them) when the model uses rotary position embeddings.

Related errors

• FA4 path does not support rotary embedding.
• FA4 does not support updating KV cache in-place.
• FA4 path does not support non-consecutive batch indices or l
• FA4 does not support updating KV cache in-place.
• FA4 path does not support non-consecutive batch indices or l

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/bfac7911bb81073a. Report an issue: GitHub.