{"record":{"id":"268a3ff76b2e134b","repo":"sgl-project/sglang","slug":"linear-attn-verify-backend-flashinfer-on-sm100","errorCode":null,"errorMessage":"--linear-attn-verify-backend flashinfer on SM100+ requires --mamba-ssm-dtype bfloat16, got {cfg.mamba_ssm_dtype!r}","messagePattern":"--linear-attn-verify-backend flashinfer on SM100\\+ requires --mamba-ssm-dtype bfloat16, got (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/server_args.py","lineNumber":6897,"sourceCode":"            and is_cuda()\n            and torch.cuda.get_device_capability()[0] >= 10\n        ):\n            raise ValueError(\n                \"--linear-attn-decode-backend flashinfer on SM100+ requires \"\n                \"--mamba-ssm-dtype bfloat16, \"\n                f\"got {cfg.mamba_ssm_dtype!r}\"\n            )\n\n        verify = cfg.linear_attn_verify_backend\n        if verify is None and decode == \"flashinfer\":\n            verify = \"flashinfer\"\n        if (\n            verify == \"flashinfer\"\n            and cfg.mamba_ssm_dtype != \"bfloat16\"\n            and is_cuda()\n            and torch.cuda.get_device_capability()[0] >= 10\n        ):\n            raise ValueError(\n                \"--linear-attn-verify-backend flashinfer on SM100+ requires \"\n                \"--mamba-ssm-dtype bfloat16, \"\n                f\"got {cfg.mamba_ssm_dtype!r}\"\n            )\n\n        # SM100+ FlashInfer GDN prefill requires CUDA 13+ (CuTe DSL kernel)\n        # for correctness and best performance.\n        prefill = cfg.linear_attn_prefill_backend or cfg.linear_attn_backend\n        cuda_version = torch.version.cuda\n        cuda_major = int(cuda_version.split(\".\")[0]) if cuda_version is not None else 0\n        if (\n            prefill == \"flashinfer\"\n            and is_cuda()\n            and torch.cuda.get_device_capability()[0] >= 10\n            and cuda_major < 13\n        ):\n            raise ValueError(\n                \"--linear-attn-prefill-backend flashinfer on SM100+ requires CUDA 13+, \"","sourceCodeStart":6879,"sourceCodeEnd":6915,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/server_args.py#L6879-L6915","documentation":"SGLang rejects --linear-attn-verify-backend flashinfer when running on an SM100+ (Blackwell) CUDA GPU unless the mamba SSM state dtype is bfloat16. The FlashInfer GDN verify kernel on SM100+ is only implemented/validated for bf16 state; other dtypes (e.g. float32) would compile or numerically fail. This check runs in the server-args resolution pipeline (_handle_linear_attn_backend) at startup.","triggerScenarios":"Setting --linear-attn-verify-backend flashinfer together with --mamba-ssm-dtype float32 (or anything != bfloat16) on a Blackwell GPU (compute capability >= 10, e.g. B200), on a CUDA build where is_cuda() is true.","commonSituations":"Users copying configs tuned for older GPUs (where fp32 mamba state was used for numerical stability) onto new B100/B200 machines; or explicitly overriding --mamba-ssm-dtype for accuracy debugging and forgetting the flashinfer verify constraint.","solutions":["Set --mamba-ssm-dtype bfloat16 (or drop the override so bf16 default applies)","If you need a non-bf16 SSM dtype on SM100+, switch the verify backend: --linear-attn-verify-backend triton (or nv_cutedsl)","If you intended a pre-Hopper/Ampere behavior, confirm torch.cuda.get_device_capability() — the check only fires on major >= 10"],"exampleFix":"# before\n--linear-attn-verify-backend flashinfer --mamba-ssm-dtype float32\n# after\n--linear-attn-verify-backend flashinfer --mamba-ssm-dtype bfloat16","handlingStrategy":"validation","validationCode":"import torch\nfrom sglang.srt.utils import is_cuda\ndef ok_flashinfer_verify(ssm_dtype):\n    return ssm_dtype == \"bfloat16\" or not (is_cuda() and torch.cuda.get_device_capability()[0] >= 10)","typeGuard":"def is_bf16_ssm(dtype: str) -> bool: return dtype == \"bfloat16\"","tryCatchPattern":null,"preventionTips":["Default to bf16 mamba state on Blackwell","Gate flashinfer verify backend selection on device capability in your launch script"],"tags":["sglang","linear-attention","flashinfer","blackwell","dtype-validation"],"backgroundTag":"gpu-arch-incompatible-config","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}