sgl-project/sglang · error · NotImplementedError

flash_attn at sgl-kernel is only supported on sm90 and above

Error message

flash_attn at sgl-kernel is only supported on sm90 and above

What it means

The sgl-kernel FA3 path (flash_attention_v3.flash_attn_with_kvcache) only runs on NVIDIA sm90+ (Hopper and later). On older GPUs _is_fa3_supported() is false and it raises NotImplementedError immediately.

Source

Thrown at python/sglang/kernels/ops/attention/flash_attention_v3.py:138

    k_descale: Optional[torch.Tensor] = None,
    v_descale: Optional[torch.Tensor] = None,
    softmax_scale=None,
    causal=False,
    window_size=(-1, -1),  # -1 means infinite context window
    attention_chunk: Optional[int] = None,
    softcap=0.0,  # 0.0 means deactivated
    rotary_interleaved=True,
    scheduler_metadata=None,
    num_splits=0,  # Can be tuned for speed
    pack_gqa=None,  # Can be tuned for speed
    only_qv=False,  # Skip K matmul when qk rope dim is 0 (requires qv)
    sm_margin=0,  # Can be tuned if some SMs are used for communication
    return_softmax_lse=False,
    sinks=None,
    out=None,
):
    if not _is_fa3_supported():
        raise NotImplementedError(
            "flash_attn at sgl-kernel is only supported on sm90 and above"
        )

    # When only_qv=True the caller may pass k_cache=None (synthetic K is
    # allocated inside the sgl-kernel wrapper). Skip the stride check in that
    # case so the rope=0 path doesn't trip the assertion.
    if k_cache is not None:
        assert k_cache.stride(-1) == 1, "k_cache must have contiguous last dimension"
    assert v_cache.stride(-1) == 1, "v_cache must have contiguous last dimension"

    return _call_fa3_kernel(
        _load_fa3_kernels()["flash_attn_with_kvcache"],
        q,
        k_cache,
        v_cache,
        k,
        v,
        qv,

View on GitHub (pinned to 0132848349)

Solutions

1. Use FA2 backend or another attention backend supported on your GPU
2. Set the server attention backend explicitly (e.g. --attention-backend fa2/flashinfer/triton)
3. Upgrade to an sm90+ GPU (H100/H200/B200) if FA3 features are required

Example fix

# before
python -m sglang.launch_server --model ... --attention-backend fa3  # on A100
# after
python -m sglang.launch_server --model ... --attention-backend fa2

import torch\nmajor, _ = torch.cuda.get_device_capability()\nif major < 9:\n    ver = 2  # FA3 needs sm90+; fall back to FA2

def fa3_supported() -> bool:\n    return torch.cuda.get_device_capability()[0] >= 9

try:\n    out = fa3.flash_attn_with_kvcache(...)\nexcept NotImplementedError:\n    out = fa2.flash_attn_with_kvcache(...)

When it happens

Trigger: Calling flash_attn_with_kvcache from flash_attention_v3 on a GPU with compute capability < 9.0 (Ampere sm80/sm86, Ada sm89, Turing).

Common situations: Running a config that selects FA3 on A100/30xx/40xx/T4 hardware; default server args choosing fa3 backend on non-Hopper machines.

Related errors

• Block sparsity + sheared bias is not supported on SM90
• qprep_bf16_fp8_sm90 requires an SM90 (Hopper) GPU
• DeepSeek-V4 flashmla_sparse_q8 prefill requires SM90 CUDA GP
• Can not import FA3 in sgl_kernel. Please check your installa
• v_cache must be provided

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/534caea3de719a66. Report an issue: GitHub.