sgl-project/sglang · error · RuntimeError

Flash attention currently only supported for compute capabil

Error message

Flash attention currently only supported for compute capability >= 80

What it means

Lightning attention's Triton/flash kernels require Ampere-or-newer tensor-core instructions; the forward explicitly checks torch.cuda.get_device_capability() and raises RuntimeError when the major version is below 8 (i.e. SM < 80, such as V100/T4).

Source

Thrown at python/sglang/kernels/ops/attention/linear/lightning_attn.py:407

    tl.store(
        O_block_ptr, qkv.to(O_block_ptr.dtype.element_ty), mask=q_index[:, None] < n
    )


class _attention(torch.autograd.Function):

    @staticmethod
    def forward(ctx, q, k, v, s, kv_history):
        # Forward pass of the lightning attention algorithm
        q = q.contiguous()
        k = k.contiguous()
        v = v.contiguous()
        s = s.contiguous()

        # Check CUDA compute capability
        capability = torch.cuda.get_device_capability()
        if capability[0] < 8:
            raise RuntimeError(
                "Flash attention currently only supported",
                "for compute capability >= 80",
            )

        # Get input dimensions
        b, h, n, d = q.shape
        e = v.shape[-1]

        # Initialize output tensor
        o = torch.empty((b, h, n, e), dtype=q.dtype, device=q.device)

        # Set block sizes
        BLOCK = 256
        NUM_BLOCK = triton.cdiv(n, BLOCK)

        CBLOCK = 32
        NUM_CBLOCK = BLOCK // CBLOCK
        assert BLOCK % CBLOCK == 0, "BLOCK must be a multiple of CBLOCK"

View on GitHub (pinned to 0132848349)

Solutions

1. Run on an SM80+ GPU (A100, A10, H100, L40S, RTX 30xx/40xx)
2. Select a non-flash attention backend (e.g. naive/triton path or a different attention implementation) for pre-Ampere devices
3. Gate the backend choice on torch.cuda.get_device_capability() at startup

Example fix

# before
backend = 'lightning_attn'  # crashes on V100
# after
major, _ = torch.cuda.get_device_capability()
backend = 'lightning_attn' if major >= 8 else 'triton'

if torch.cuda.get_device_capability()[0] < 8:
    attn_backend = 'naive'  # or any non-flash path
else:
    attn_backend = 'lightning_attn'

except RuntimeError as e: if 'compute capability' in str(e): use fallback attention backend

When it happens

Trigger: Calling lightning_attn forward (lightning_attn.py:forward) on a pre-Ampere GPU (compute capability 7.x or lower).

Common situations: Running or unit-testing lightning attention models on V100, GTX/RTX Turing, or older datacenter GPUs; defaulting to the lightning_attn backend on a heterogeneous cluster with older cards.

Related errors

• qprep_bf16_fp8_sm90 requires an SM90 (Hopper) GPU
• Cannot find NVIDIA Math-DX (cuBLASDx) headers. Install the `
• flash_attn at sgl-kernel is only supported on sm90 and above
• {name}_block_cnt and {name}_block_idx must be on the same de
• {name}_block tensors must live on CUDA

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/4b4ac95ab0eac0f2. Report an issue: GitHub.