vllm-project/vllm · error · ValueError

Stochastic rounding for Mamba cache requires the SSM cache t

Error message

Stochastic rounding for Mamba cache requires the SSM cache to be float16. Please set it explicitly, by specifying `--mamba-ssm-cache-dtype float16`, or disable stochastic rounding by not specifying `--enable-mamba-cache-stochastic-rounding`.

What it means

vLLM's Mamba/SSM cache stochastic rounding only works when the SSM state cache is stored in float16, because the rounding kernel operates on 16-bit values. During VllmConfig validation, enabling `--enable-mamba-cache-stochastic-rounding` with a `mamba_ssm_cache_dtype` other than 'float16' raises this ValueError at startup.

Source

Thrown at vllm/config/vllm.py:1109

            if (
                self.kv_transfer_config is not None
                and self.kv_transfer_config.is_kv_transfer_instance
            ):
                raise ValueError(
                    "--enable-return-routed-experts is incompatible with KV "
                    "connectors (PD disaggregation, KV cache offload)."
                )

        self._verify_sampling_replay_config()

        if self.lora_config is not None:
            self.lora_config.verify_with_model_config(self.model_config)

        if (
            self.mamba_config.enable_stochastic_rounding
            and self.cache_config.mamba_ssm_cache_dtype != "float16"
        ):
            raise ValueError(
                "Stochastic rounding for Mamba cache requires "
                "the SSM cache to be float16. Please set it explicitly, "
                "by specifying `--mamba-ssm-cache-dtype float16`, or disable "
                "stochastic rounding by not specifying "
                "`--enable-mamba-cache-stochastic-rounding`."
            )

        if self.quant_config is None and self.model_config is not None:
            self.quant_config = VllmConfig._get_quantization_config(
                self.model_config, self.load_config
            )

        if (
            self.quant_config is not None
            and self.model_config is not None
            and hasattr(self.quant_config, "use_deep_gemm")
            and self.quant_config.use_deep_gemm is None
        ):

View on GitHub (pinned to c794754062)

Solutions

Add `--mamba-ssm-cache-dtype float16` to the launch command so the SSM cache matches the stochastic rounding requirement.
Or remove `--enable-mamba-cache-stochastic-rounding` to disable the feature entirely.
If you need non-fp16 SSM caches, do not use stochastic rounding; rely on standard cache storage instead.

Example fix

# before
vllm serve mistralai/Mamba-Codestral-7B-v0.1 \
  --enable-mamba-cache-stochastic-rounding

# after
vllm serve mistralai/Mamba-Codestral-7B-v0.1 \
  --enable-mamba-cache-stochastic-rounding \
  --mamba-ssm-cache-dtype float16

Defensive patterns

Strategy: validation

Validate before calling

# before constructing EngineArgs / VllmConfig
if args.enable_mamba_cache_stochastic_rounding and mamba_ssm_cache_dtype not in (None, "float16"):
    raise SystemExit("set --mamba-ssm-cache-dtype float16 or drop --enable-mamba-cache-stochastic-rounding")

Try / catch

try:
    llm = LLM(**engine_args)
except ValueError as e:
    if "Stochastic rounding for Mamba" in str(e):
        log_and_retry_with(dtype="float16")
    else:
        raise

Prevention

Treat stochastic-rounding and ssm-cache-dtype as one unit in launch templates
Add a config lint step that pairs the two flags before deployment

When it happens

Trigger: Starting vLLM with `--enable-mamba-cache-stochastic-rounding` while `--mamba-ssm-cache-dtype` is unset (defaults to something other than float16, e.g. 'auto'/bfloat16) or explicitly set to a non-float16 dtype, on a Mamba/Hybrid-Mamba model.

Common situations: Users enabling stochastic rounding for memory savings on Mamba models (e.g. Falcon-H1, Mamba2, Nemotron-H) without pinning the SSM cache dtype; or previously setting `--mamba-ssm-cache-dtype bfloat16` and later adding the rounding flag.

Related errors

AI-assisted analysis of vllm-project/vllm@c794754062 (2026-08-14). Data as JSON: /api/errors/f69edfd0ddb4a871. Report an issue: GitHub.