sgl-project/sglang · error · ValueError

--linear-attn-decode-backend flashinfer on SM100+ requires -

Error message

--linear-attn-decode-backend flashinfer on SM100+ requires --mamba-ssm-dtype bfloat16, got {cfg.mamba_ssm_dtype!r}

What it means

The FlashInfer linear-attention decode kernel on SM100+ (Blackwell) GPUs is only implemented for bfloat16 SSM state. If --linear-attn-decode-backend flashinfer is combined with any other --mamba-ssm-dtype on a CUDA device with compute capability >= 10, ServerArgs resolution rejects it.

Source

Thrown at python/sglang/srt/server_args.py:6882

                    "--linear-attn-prefill-backend flashkda (decode stays on triton)."
                )
            self._declare(
                "_handle_linear_attn_backend",
                linear_attn_decode_backend="triton",
            )
            decode = "triton"
            logger.info(
                "FlashKDA is prefill-only; using triton for KDA decode "
                "(FlashKDA stays on prefill)."
            )

        if (
            decode == "flashinfer"
            and cfg.mamba_ssm_dtype != "bfloat16"
            and is_cuda()
            and torch.cuda.get_device_capability()[0] >= 10
        ):
            raise ValueError(
                "--linear-attn-decode-backend flashinfer on SM100+ requires "
                "--mamba-ssm-dtype bfloat16, "
                f"got {cfg.mamba_ssm_dtype!r}"
            )

        verify = cfg.linear_attn_verify_backend
        if verify is None and decode == "flashinfer":
            verify = "flashinfer"
        if (
            verify == "flashinfer"
            and cfg.mamba_ssm_dtype != "bfloat16"
            and is_cuda()
            and torch.cuda.get_device_capability()[0] >= 10
        ):
            raise ValueError(
                "--linear-attn-verify-backend flashinfer on SM100+ requires "
                "--mamba-ssm-dtype bfloat16, "
                f"got {cfg.mamba_ssm_dtype!r}"

View on GitHub (pinned to 0132848349)

Solutions

1. Add --mamba-ssm-dtype bfloat16 to the launch command
2. Switch to a decode backend without the bf16 requirement (e.g. triton) if you must keep the current dtype

Example fix

# before (on SM100+)
--linear-attn-decode-backend flashinfer --mamba-ssm-dtype float16
# after
--linear-attn-decode-backend flashinfer --mamba-ssm-dtype bfloat16

import torch
if args.linear_attn_decode_backend == "flashinfer" and torch.cuda.is_available():
    if torch.cuda.get_device_capability()[0] >= 10:
        assert args.mamba_ssm_dtype == "bfloat16", "flashinfer decode on SM100+ requires bf16 SSM dtype"

null

When it happens

Trigger: Launching with --linear-attn-decode-backend flashinfer and --mamba-ssm-dtype not bfloat16 (e.g. float16, float32) on a Blackwell GPU (torch.cuda.get_device_capability()[0] >= 10). Non-CUDA platforms or SM < 10 do not trigger this check.

Common situations: Reusing a float16 SSM config (tuned for stochastic rounding or Hopper) on a B200 machine while switching the decode backend to flashinfer; defaults drifting after a version upgrade.

Related errors

• FlashInfer GDN prefill is not supported with --enable-determ
• FlashInfer KDA kernel (recurrent_kda) is not available. Requ
• --linear-attn-verify-backend flashinfer on SM100+ requires -
• --linear-attn-prefill-backend flashinfer on SM100+ requires
• Unsupported CUTLASS scalar type for A/B: {cutlass_type!r}

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/07b29386803dd0e6. Report an issue: GitHub.