sgl-project/sglang · error · ValueError

Stochastic rounding for the Mamba SSM cache with --mamba-bac

Error message

Stochastic rounding for the Mamba SSM cache with --mamba-backend triton requires SM100 with CUDA >= 12.8 because it uses the cvt.rs.f16x2.f32 PTX instruction. On H100/SM90, run with --mamba-backend flashinfer --mamba-ssm-dtype float16, or disable --enable-mamba-cache-stochastic-rounding.

What it means

The triton Mamba backend's stochastic rounding uses the cvt.rs.f16x2.f32 PTX instruction, which only exists on SM100 (Blackwell) with CUDA >= 12.8. On older GPUs (H100/SM90 etc.) the combination --mamba-backend triton + --enable-mamba-cache-stochastic-rounding is rejected.

Source

Thrown at python/sglang/srt/server_args.py:6772

                f"integer, got {cfg.mamba_max_states_per_path}."
            )

        if cfg.enable_mamba_cache_stochastic_rounding:
            if cfg.mamba_ssm_dtype != "float16":
                raise ValueError(
                    "Stochastic rounding for the Mamba SSM cache requires "
                    f"--mamba-ssm-dtype float16, got {cfg.mamba_ssm_dtype!r}. "
                    "Run with --mamba-ssm-dtype float16 or disable "
                    "--enable-mamba-cache-stochastic-rounding."
                )
            if not is_cuda():
                raise ValueError(
                    "Stochastic rounding for the Mamba SSM cache is only "
                    "supported on NVIDIA CUDA platforms. Disable "
                    "--enable-mamba-cache-stochastic-rounding on this platform."
                )
            if cfg.mamba_backend == "triton" and not is_sm100_supported():
                raise ValueError(
                    "Stochastic rounding for the Mamba SSM cache with "
                    "--mamba-backend triton requires SM100 with CUDA >= 12.8 "
                    "because it uses the cvt.rs.f16x2.f32 PTX instruction. On "
                    "H100/SM90, run with --mamba-backend flashinfer "
                    "--mamba-ssm-dtype float16, or disable "
                    "--enable-mamba-cache-stochastic-rounding."
                )

        if cfg.mamba_backend == "flashinfer":
            flashinfer_error = (
                "FlashInfer mamba module not available, please check the "
                "FlashInfer installation."
            )
            if cfg.enable_mamba_cache_stochastic_rounding:
                flashinfer_error += (
                    " Stochastic rounding with --mamba-backend flashinfer "
                    "requires FlashInfer Mamba and --mamba-ssm-dtype float16."
                )

View on GitHub (pinned to 0132848349)

Solutions

1. On SM90: switch to --mamba-backend flashinfer --mamba-ssm-dtype float16 and keep stochastic rounding
2. Disable --enable-mamba-cache-stochastic-rounding and keep triton
3. Upgrade the CUDA toolkit to >= 12.8 if you are actually on SM100 hardware

Example fix

# before (on H100)
--mamba-backend triton --enable-mamba-cache-stochastic-rounding --mamba-ssm-dtype float16
# after (on H100)
--mamba-backend flashinfer --enable-mamba-cache-stochastic-rounding --mamba-ssm-dtype float16

import torch
from sglang.srt.utils import is_sm100_supported
if args.enable_mamba_cache_stochastic_rounding and args.mamba_backend == "triton":
    assert is_sm100_supported(), "triton stochastic rounding needs SM100 + CUDA>=12.8"

null

When it happens

Trigger: Launching with --enable-mamba-cache-stochastic-rounding --mamba-backend triton on a GPU where is_sm100_supported() is False (H100, A100, or CUDA < 12.8).

Common situations: A config tuned on a B200/Blackwell box reused on an H100 cluster; older CUDA toolkit (12.4) installed so is_sm100_supported() is False even on new hardware.

Related errors

• Stochastic rounding for the Mamba SSM cache requires --mamba
• `mixed_qkv` must be a 2D tensor (got ndim={mixed_qkv.ndim}).
• cutedsl_bf16_gemm requires an SM10x GPU
• {fn_name}: dst entry dims (dims {entry_start_dim}..{dst.ndim
• nvfp4_gemm_swiglu_nvfp4_quant requires SM100, got SM{major}{

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/df746c619774e978. Report an issue: GitHub.