sgl-project/sglang · error · ValueError

--linear-attn-verify-backend flashinfer on SM100+ requires -

Error message

--linear-attn-verify-backend flashinfer on SM100+ requires --mamba-ssm-dtype bfloat16, got {cfg.mamba_ssm_dtype!r}

What it means

SGLang rejects --linear-attn-verify-backend flashinfer when running on an SM100+ (Blackwell) CUDA GPU unless the mamba SSM state dtype is bfloat16. The FlashInfer GDN verify kernel on SM100+ is only implemented/validated for bf16 state; other dtypes (e.g. float32) would compile or numerically fail. This check runs in the server-args resolution pipeline (_handle_linear_attn_backend) at startup.

Source

Thrown at python/sglang/srt/server_args.py:6897

            and is_cuda()
            and torch.cuda.get_device_capability()[0] >= 10
        ):
            raise ValueError(
                "--linear-attn-decode-backend flashinfer on SM100+ requires "
                "--mamba-ssm-dtype bfloat16, "
                f"got {cfg.mamba_ssm_dtype!r}"
            )

        verify = cfg.linear_attn_verify_backend
        if verify is None and decode == "flashinfer":
            verify = "flashinfer"
        if (
            verify == "flashinfer"
            and cfg.mamba_ssm_dtype != "bfloat16"
            and is_cuda()
            and torch.cuda.get_device_capability()[0] >= 10
        ):
            raise ValueError(
                "--linear-attn-verify-backend flashinfer on SM100+ requires "
                "--mamba-ssm-dtype bfloat16, "
                f"got {cfg.mamba_ssm_dtype!r}"
            )

        # SM100+ FlashInfer GDN prefill requires CUDA 13+ (CuTe DSL kernel)
        # for correctness and best performance.
        prefill = cfg.linear_attn_prefill_backend or cfg.linear_attn_backend
        cuda_version = torch.version.cuda
        cuda_major = int(cuda_version.split(".")[0]) if cuda_version is not None else 0
        if (
            prefill == "flashinfer"
            and is_cuda()
            and torch.cuda.get_device_capability()[0] >= 10
            and cuda_major < 13
        ):
            raise ValueError(
                "--linear-attn-prefill-backend flashinfer on SM100+ requires CUDA 13+, "

View on GitHub (pinned to 0132848349)

Solutions

  1. Set --mamba-ssm-dtype bfloat16 (or drop the override so bf16 default applies)
  2. If you need a non-bf16 SSM dtype on SM100+, switch the verify backend: --linear-attn-verify-backend triton (or nv_cutedsl)
  3. If you intended a pre-Hopper/Ampere behavior, confirm torch.cuda.get_device_capability() — the check only fires on major >= 10

Example fix

# before
--linear-attn-verify-backend flashinfer --mamba-ssm-dtype float32
# after
--linear-attn-verify-backend flashinfer --mamba-ssm-dtype bfloat16
Defensive patterns

Strategy: validation

Validate before calling

import torch
from sglang.srt.utils import is_cuda
def ok_flashinfer_verify(ssm_dtype):
    return ssm_dtype == "bfloat16" or not (is_cuda() and torch.cuda.get_device_capability()[0] >= 10)

Type guard

def is_bf16_ssm(dtype: str) -> bool: return dtype == "bfloat16"

Prevention

When it happens

Trigger: Setting --linear-attn-verify-backend flashinfer together with --mamba-ssm-dtype float32 (or anything != bfloat16) on a Blackwell GPU (compute capability >= 10, e.g. B200), on a CUDA build where is_cuda() is true.

Common situations: Users copying configs tuned for older GPUs (where fp32 mamba state was used for numerical stability) onto new B100/B200 machines; or explicitly overriding --mamba-ssm-dtype for accuracy debugging and forgetting the flashinfer verify constraint.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/268a3ff76b2e134b. Report an issue: GitHub.