sgl-project/sglang · error · ValueError

Stochastic rounding for the Mamba SSM cache is only supporte

Error message

Stochastic rounding for the Mamba SSM cache is only supported on NVIDIA CUDA platforms. Disable --enable-mamba-cache-stochastic-rounding on this platform.

What it means

The stochastic-rounding Mamba SSM cache kernels are CUDA-only; the is_cuda() platform check in _handle_mamba_backend fails on ROCm, CPU, or any non-NVIDIA build, and the flag combination is rejected at startup rather than crashing later inside a kernel launch.

Source

Thrown at python/sglang/srt/server_args.py:6766

        if cfg.mamba_cache_philox_rounds < 0:
            raise ValueError("--mamba-cache-philox-rounds must be non-negative.")

        if cfg.mamba_max_states_per_path == 0 or cfg.mamba_max_states_per_path < -1:
            raise ValueError(
                "--mamba-max-states-per-path must be -1 (unlimited) or a positive "
                f"integer, got {cfg.mamba_max_states_per_path}."
            )

        if cfg.enable_mamba_cache_stochastic_rounding:
            if cfg.mamba_ssm_dtype != "float16":
                raise ValueError(
                    "Stochastic rounding for the Mamba SSM cache requires "
                    f"--mamba-ssm-dtype float16, got {cfg.mamba_ssm_dtype!r}. "
                    "Run with --mamba-ssm-dtype float16 or disable "
                    "--enable-mamba-cache-stochastic-rounding."
                )
            if not is_cuda():
                raise ValueError(
                    "Stochastic rounding for the Mamba SSM cache is only "
                    "supported on NVIDIA CUDA platforms. Disable "
                    "--enable-mamba-cache-stochastic-rounding on this platform."
                )
            if cfg.mamba_backend == "triton" and not is_sm100_supported():
                raise ValueError(
                    "Stochastic rounding for the Mamba SSM cache with "
                    "--mamba-backend triton requires SM100 with CUDA >= 12.8 "
                    "because it uses the cvt.rs.f16x2.f32 PTX instruction. On "
                    "H100/SM90, run with --mamba-backend flashinfer "
                    "--mamba-ssm-dtype float16, or disable "
                    "--enable-mamba-cache-stochastic-rounding."
                )

        if cfg.mamba_backend == "flashinfer":
            flashinfer_error = (
                "FlashInfer mamba module not available, please check the "
                "FlashInfer installation."

View on GitHub (pinned to 0132848349)

Solutions

1. Remove --enable-mamba-cache-stochastic-rounding from the launch command on this platform
2. Run on an NVIDIA CUDA machine if stochastic rounding is required
3. Gate the flag in your launcher: only add it when torch.cuda.is_available() and the device is NVIDIA

Example fix

# before
python -m sglang.launch_server --enable-mamba-cache-stochastic-rounding ...  # on ROCm
# after
python -m sglang.launch_server ...  # flag omitted on non-CUDA platforms

import torch
if args.enable_mamba_cache_stochastic_rounding:
    assert torch.cuda.is_available(), "stochastic rounding requires NVIDIA CUDA"

null

When it happens

Trigger: Launching with --enable-mamba-cache-stochastic-rounding on a machine where is_cuda() returns False (AMD ROCm, CPU-only node, non-CUDA build of sglang).

Common situations: Running the same launch script on an AMD MI300x or CPU-only dev box that worked on an NVIDIA cluster; CI jobs that don't gate the flag on platform.

Related errors

• --mamba-max-states-per-path must be -1 (unlimited) or a posi
• Stochastic rounding for the Mamba SSM cache requires --mamba
• --enable-int8-mamba-checkpoint is not supported together wit
• --enable-int8-mamba-checkpoint only supports the built-in ma
• MLX auxiliary-state radix cache does not support enable_mamb

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/b72816f1165dd2d3. Report an issue: GitHub.