{"record":{"id":"f69edfd0ddb4a871","repo":"vllm-project/vllm","slug":"stochastic-rounding-for-mamba-cache-requires-the-s","errorCode":null,"errorMessage":"Stochastic rounding for Mamba cache requires the SSM cache to be float16. Please set it explicitly, by specifying `--mamba-ssm-cache-dtype float16`, or disable stochastic rounding by not specifying `--enable-mamba-cache-stochastic-rounding`.","messagePattern":"Stochastic rounding for Mamba cache requires the SSM cache to be float16\\. Please set it explicitly, by specifying `--mamba-ssm-cache-dtype float16`, or disable stochastic rounding by not specifying `--enable-mamba-cache-stochastic-rounding`\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"vllm/config/vllm.py","lineNumber":1109,"sourceCode":"            if (\n                self.kv_transfer_config is not None\n                and self.kv_transfer_config.is_kv_transfer_instance\n            ):\n                raise ValueError(\n                    \"--enable-return-routed-experts is incompatible with KV \"\n                    \"connectors (PD disaggregation, KV cache offload).\"\n                )\n\n        self._verify_sampling_replay_config()\n\n        if self.lora_config is not None:\n            self.lora_config.verify_with_model_config(self.model_config)\n\n        if (\n            self.mamba_config.enable_stochastic_rounding\n            and self.cache_config.mamba_ssm_cache_dtype != \"float16\"\n        ):\n            raise ValueError(\n                \"Stochastic rounding for Mamba cache requires \"\n                \"the SSM cache to be float16. Please set it explicitly, \"\n                \"by specifying `--mamba-ssm-cache-dtype float16`, or disable \"\n                \"stochastic rounding by not specifying \"\n                \"`--enable-mamba-cache-stochastic-rounding`.\"\n            )\n\n        if self.quant_config is None and self.model_config is not None:\n            self.quant_config = VllmConfig._get_quantization_config(\n                self.model_config, self.load_config\n            )\n\n        if (\n            self.quant_config is not None\n            and self.model_config is not None\n            and hasattr(self.quant_config, \"use_deep_gemm\")\n            and self.quant_config.use_deep_gemm is None\n        ):","sourceCodeStart":1091,"sourceCodeEnd":1127,"githubUrl":"https://github.com/vllm-project/vllm/blob/c794754062d49a8fdb63ab3c5215b488b865030c/vllm/config/vllm.py#L1091-L1127","documentation":"vLLM's Mamba/SSM cache stochastic rounding only works when the SSM state cache is stored in float16, because the rounding kernel operates on 16-bit values. During VllmConfig validation, enabling `--enable-mamba-cache-stochastic-rounding` with a `mamba_ssm_cache_dtype` other than 'float16' raises this ValueError at startup.","triggerScenarios":"Starting vLLM with `--enable-mamba-cache-stochastic-rounding` while `--mamba-ssm-cache-dtype` is unset (defaults to something other than float16, e.g. 'auto'/bfloat16) or explicitly set to a non-float16 dtype, on a Mamba/Hybrid-Mamba model.","commonSituations":"Users enabling stochastic rounding for memory savings on Mamba models (e.g. Falcon-H1, Mamba2, Nemotron-H) without pinning the SSM cache dtype; or previously setting `--mamba-ssm-cache-dtype bfloat16` and later adding the rounding flag.","solutions":["Add `--mamba-ssm-cache-dtype float16` to the launch command so the SSM cache matches the stochastic rounding requirement.","Or remove `--enable-mamba-cache-stochastic-rounding` to disable the feature entirely.","If you need non-fp16 SSM caches, do not use stochastic rounding; rely on standard cache storage instead."],"exampleFix":"# before\nvllm serve mistralai/Mamba-Codestral-7B-v0.1 \\\n  --enable-mamba-cache-stochastic-rounding\n\n# after\nvllm serve mistralai/Mamba-Codestral-7B-v0.1 \\\n  --enable-mamba-cache-stochastic-rounding \\\n  --mamba-ssm-cache-dtype float16","handlingStrategy":"validation","validationCode":"# before constructing EngineArgs / VllmConfig\nif args.enable_mamba_cache_stochastic_rounding and mamba_ssm_cache_dtype not in (None, \"float16\"):\n    raise SystemExit(\"set --mamba-ssm-cache-dtype float16 or drop --enable-mamba-cache-stochastic-rounding\")","typeGuard":null,"tryCatchPattern":"try:\n    llm = LLM(**engine_args)\nexcept ValueError as e:\n    if \"Stochastic rounding for Mamba\" in str(e):\n        log_and_retry_with(dtype=\"float16\")\n    else:\n        raise","preventionTips":["Treat stochastic-rounding and ssm-cache-dtype as one unit in launch templates","Add a config lint step that pairs the two flags before deployment"],"tags":["mamba","cache-dtype","stochastic-rounding","startup-config"],"backgroundTag":null,"analyzedSha":"c794754062d49a8fdb63ab3c5215b488b865030c","analyzedAt":"2026-08-14T21:17:39.825Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}