sgl-project/sglang · error · RuntimeError

score_mod is only supported by the FA4 backend.

Error message

score_mod is only supported by the FA4 backend.

What it means

forward_extend raises when a score_mod (flexible-attention score modification, e.g. soft-cap or custom bias function) is requested but the active FA implementation is not FA4, since only FA4 exposes the score_mod hook.

Source

Thrown at python/sglang/srt/layers/attention/flashattention_backend.py:1249

        k: torch.Tensor,
        v: torch.Tensor,
        layer: RadixAttention,
        forward_batch: ForwardBatch,
        save_kv_cache=True,
        # For multi-head latent attention
        q_rope: Optional[torch.Tensor] = None,
        k_rope: Optional[torch.Tensor] = None,
        sinks: Optional[torch.Tensor] = None,
        q_descale: Optional[torch.Tensor] = None,
        k_descale: Optional[torch.Tensor] = None,
        v_descale: Optional[torch.Tensor] = None,
        score_mod=None,
        aux_tensors=None,
        rel_bias=None,
        rel_bias_event=None,
    ):
        if score_mod is not None and self.fa_impl_ver != 4:
            raise RuntimeError("score_mod is only supported by the FA4 backend.")
        is_cp_mode = (
            forward_batch.forward_mode.is_context_parallel_extend()
            and forward_batch.attn_cp_metadata is not None
            and self.attn_cp_size > 1
        )

        if k is not None:
            assert v is not None

            if save_kv_cache and not self.fa_skip_kv_cache:
                cache_loc = (
                    forward_batch.out_cache_loc
                    if not layer.is_cross_attention
                    else forward_batch.encoder_out_cache_loc
                )
                if self.use_mla:
                    if is_cp_v2_active(forward_batch):
                        # CP-v2: k/k_rope are rank-local; the strategy gathers

View on GitHub (pinned to 0132848349)

Solutions

1. Switch to the FA4 backend (Blackwell GPU + FA4 install, force version 4)
2. If the model supports it, use its non-score_mod fallback (e.g. separate softcap flag) so score_mod stays None
3. Use a different attention backend that natively supports the needed modification (e.g. FlashInfer)

Example fix

# before
SGLANG_FLASHATTENTION_VERSION=3 python -m sglang.launch_server --model gemma-3-27b ...
# after
SGLANG_FLASHATTENTION_VERSION=4 python -m sglang.launch_server --model gemma-3-27b ...

assert score_mod is None or fa_impl_ver == 4, 'score_mod requires FA4 backend'

When it happens

Trigger: A model layer (e.g. Gemma-style logit softcapping via score_mod or a custom attention variant) passes score_mod while running on FA2/FA3.

Common situations: Running a score_mod-dependent model with SGLANG_FLASHATTENTION_VERSION=3 or 2, or on hardware where FA4 is unavailable so startup silently picked FA3.

Related errors

• rel_bias (sheared bias) is only supported by the FA4 backend
• MXFP8 KV cache requires the FA4 backend.
• MXFP8 KV cache requires per-token Q scales (q_descale) from
• VisionFlash4Attention is only available for cuda
• Unknown serve backend {name!r}. Available values: {available

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/e65c1d665ae1a4e8. Report an issue: GitHub.