sgl-project/sglang · error · ValueError

Stochastic rounding for the Mamba SSM cache requires --mamba

Error message

Stochastic rounding for the Mamba SSM cache requires --mamba-ssm-dtype float16, got {cfg.mamba_ssm_dtype!r}. Run with --mamba-ssm-dtype float16 or disable --enable-mamba-cache-stochastic-rounding.

What it means

Stochastic rounding for the Mamba SSM cache (enabled via --enable-mamba-cache-stochastic-rounding) is only implemented for the float16 SSM state dtype. The kernel that performs stochastic rounding operates on fp16 state, so any other --mamba-ssm-dtype (e.g. bfloat16, float32) is rejected up front.

Source

Thrown at python/sglang/srt/server_args.py:6759

    def _handle_grammar_backend(self):
        cfg = resolving_view(self)
        if cfg.grammar_backend is None:
            self._declare("_handle_grammar_backend", grammar_backend="xgrammar")

    def _handle_mamba_backend(self):
        cfg = resolving_view(self)
        if cfg.mamba_cache_philox_rounds < 0:
            raise ValueError("--mamba-cache-philox-rounds must be non-negative.")

        if cfg.mamba_max_states_per_path == 0 or cfg.mamba_max_states_per_path < -1:
            raise ValueError(
                "--mamba-max-states-per-path must be -1 (unlimited) or a positive "
                f"integer, got {cfg.mamba_max_states_per_path}."
            )

        if cfg.enable_mamba_cache_stochastic_rounding:
            if cfg.mamba_ssm_dtype != "float16":
                raise ValueError(
                    "Stochastic rounding for the Mamba SSM cache requires "
                    f"--mamba-ssm-dtype float16, got {cfg.mamba_ssm_dtype!r}. "
                    "Run with --mamba-ssm-dtype float16 or disable "
                    "--enable-mamba-cache-stochastic-rounding."
                )
            if not is_cuda():
                raise ValueError(
                    "Stochastic rounding for the Mamba SSM cache is only "
                    "supported on NVIDIA CUDA platforms. Disable "
                    "--enable-mamba-cache-stochastic-rounding on this platform."
                )
            if cfg.mamba_backend == "triton" and not is_sm100_supported():
                raise ValueError(
                    "Stochastic rounding for the Mamba SSM cache with "
                    "--mamba-backend triton requires SM100 with CUDA >= 12.8 "
                    "because it uses the cvt.rs.f16x2.f32 PTX instruction. On "
                    "H100/SM90, run with --mamba-backend flashinfer "
                    "--mamba-ssm-dtype float16, or disable "

View on GitHub (pinned to 0132848349)

Solutions

Add --mamba-ssm-dtype float16 to the launch command
Disable stochastic rounding by dropping --enable-mamba-cache-stochastic-rounding

Example fix

# before
--enable-mamba-cache-stochastic-rounding --mamba-ssm-dtype bfloat16
# after
--enable-mamba-cache-stochastic-rounding --mamba-ssm-dtype float16

Defensive patterns

Strategy: validation

Validate before calling

sr = args.enable_mamba_cache_stochastic_rounding
if sr:
    assert args.mamba_ssm_dtype == "float16", "stochastic rounding requires --mamba-ssm-dtype float16"

Type guard

null

Prevention

Keep a compatibility table of flag pairs; stochastic rounding is coupled to float16 state
Set both flags together in one config macro instead of independently

When it happens

Trigger: Starting the server with --enable-mamba-cache-stochastic-rounding while --mamba-ssm-dtype is anything other than float16. Checked inside _handle_mamba_backend during ServerArgs resolution, so it aborts startup immediately.

Common situations: Users copying a bfloat16 SSM config (common for quality reasons) and layering the stochastic-rounding optimization on top; or defaults changing between versions so mamba_ssm_dtype is no longer float16 when the rounding flag is set.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/4e60d1276552a344. Report an issue: GitHub.