sgl-project/sglang · error · ValueError

--linear-attn-prefill-backend flashinfer on SM100+ requires

Error message

--linear-attn-prefill-backend flashinfer on SM100+ requires CUDA 13+, got CUDA {cuda_version or 'unknown'}

What it means

SGLang rejects --linear-attn-prefill-backend flashinfer on SM100+ (Blackwell) GPUs when the installed CUDA toolkit major version is below 13. The SM100+ FlashInfer GDN prefill path uses a CuTe DSL kernel that requires CUDA 13+ to build/run. Raised during server-args resolution at startup.

Source

Thrown at python/sglang/srt/server_args.py:6914

        ):
            raise ValueError(
                "--linear-attn-verify-backend flashinfer on SM100+ requires "
                "--mamba-ssm-dtype bfloat16, "
                f"got {cfg.mamba_ssm_dtype!r}"
            )

        # SM100+ FlashInfer GDN prefill requires CUDA 13+ (CuTe DSL kernel)
        # for correctness and best performance.
        prefill = cfg.linear_attn_prefill_backend or cfg.linear_attn_backend
        cuda_version = torch.version.cuda
        cuda_major = int(cuda_version.split(".")[0]) if cuda_version is not None else 0
        if (
            prefill == "flashinfer"
            and is_cuda()
            and torch.cuda.get_device_capability()[0] >= 10
            and cuda_major < 13
        ):
            raise ValueError(
                "--linear-attn-prefill-backend flashinfer on SM100+ requires CUDA 13+, "
                f"got CUDA {cuda_version or 'unknown'}"
            )

        # ReplaySSM buffered decode guards. Runs on Triton, or Helion for KDA.
        # cuda-graph is supported (slice 1b: CUDA-graph-safe static
        # write-cursor buffers). The RADIX prefix cache is now supported (slice
        # 2b: the decode kernel force-flushes the ring into temporal[slot] on
        # the radix track boundary `seq_lens % mamba_track_interval == 0`, and
        # the COW copy-into-slot path resets the ring cursor) -- so the
        # --disable-radix-cache requirement is dropped.
        #
        # Slice 2b only wires the no_buffer mamba scheduler strategy (the
        # default). The extra_buffer strategy donates the track snapshot via
        # `donate_mamba_ping_pong_slot` with a separate ping-pong slot swap that
        # does NOT route through MambaPool.copy_from, so the ReplaySSM ring
        # cursor of the donated/kept slot would not be reset there. Handling
        # that donation path is a follow-up; for now require no_buffer.

View on GitHub (pinned to 0132848349)

Solutions

Upgrade to a CUDA 13+ build/container (e.g. pytorch:nightly or ngc image with CUDA 13) and restart
Or switch prefill backend on CUDA 12: --linear-attn-prefill-backend triton
Verify with python -c "import torch; print(torch.version.cuda, torch.cuda.get_device_capability())"

Example fix

# before (CUDA 12.x container)
--linear-attn-prefill-backend flashinfer
# after
# option A: run under CUDA 13+ image
# option B: --linear-attn-prefill-backend triton

Defensive patterns

Strategy: validation

Validate before calling

import torch
def cuda_major():
    v = torch.version.cuda
    return int(v.split('.')[0]) if v else 0
def ok_flashinfer_prefill():
    return cuda_major() >= 13 or (not torch.cuda.is_available()) or torch.cuda.get_device_capability()[0] < 10

Prevention

Pin deployment containers to CUDA 13+ on Blackwell fleets
Assert torch.version.cuda >= 13 in smoke tests before server launch

When it happens

Trigger: --linear-attn-prefill-backend flashinfer on a GPU with compute capability >= 10 while torch.version.cuda / nvcc reports CUDA 12.x (e.g. standard CUDA 12.4/12.6 containers on B200).

Common situations: Running Blackwell hardware inside the common CUDA 12.x NGC/pytorch images; upgrading GPUs without upgrading the CUDA toolkit/container; CI images pinned to CUDA 12.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/71e80fc13aee7f40. Report an issue: GitHub.