vllm-project/vllm · error · ValueError

--use-replayssm does not support speculative decoding

Error message

--use-replayssm does not support speculative decoding

What it means

Raised by VllmConfig.validate_mamba_cached_kernel when ReplaySSM (--use-replayssm) is combined with speculative decoding (num_speculative_tokens > 0). The replayed SSM state ring cannot be kept consistent when draft tokens are replayed, so vLLM rejects the combination during config validation.

Source

Thrown at vllm/config/vllm.py:2576

    @model_validator(mode="after")
    def validate_mamba_cached_kernel(self) -> "VllmConfig":
        if not self.cache_config.use_replayssm:
            return self
        # ReplaySSM adds a 3-tensor ring to the mamba state; only models that
        # opt in (supports_replayssm) build a consistent shape on both the layer
        # and config paths. Reject others so the mamba page size cannot desync.
        if self.model_config is not None and not self.model_config.supports_replayssm:
            raise ValueError(
                "--use-replayssm is only supported for Nemotron-H models "
                f"(got architecture {self.model_config.architecture!r})"
            )
        if self.cache_config.mamba_cache_mode == "all":
            raise ValueError(
                "--use-replayssm supports prefix caching only in align mode; "
                "pass --mamba-cache-mode align"
            )
        if self.num_speculative_tokens > 0:
            raise ValueError("--use-replayssm does not support speculative decoding")
        if self.mamba_config.backend != MambaBackendEnum.TRITON:
            raise ValueError("--use-replayssm requires --mamba-backend triton")
        if (
            self.kv_transfer_config is not None
            and self.kv_transfer_config.is_kv_transfer_instance
        ):
            raise ValueError(
                "--use-replayssm is incompatible with KV connectors "
                "(P/D disaggregation, KV cache offload)"
            )
        return self


_current_vllm_config: VllmConfig | None = None
_current_prefix: str | None = None


@contextmanager

View on GitHub (pinned to c794754062)

Solutions

Disable speculative decoding: remove/zero the spec-decode method and num_speculative_tokens from the launch config.
If spec decode is required, drop --use-replayssm.

Example fix

# before
vllm serve model --use-replayssm \
  --speculative-config '{"method":"ngram","num_speculative_tokens":3}'
# after
vllm serve model --use-replayssm

Defensive patterns

Strategy: validation

Validate before calling

if cfg.num_speculative_tokens > 0 and cfg.cache_config.use_replayssm:
    raise SystemExit("choose either --use-replayssm or speculative decoding, not both")

Type guard

def spec_decode_compatible(cfg) -> bool:
    return not cfg.cache_config.use_replayssm or cfg.num_speculative_tokens == 0

Try / catch

try:
    engine = LLM(**args)
except ValueError as e:
    if "speculative" in str(e) and "replayssm" in str(e):
        args.pop("speculative_config"); retry_launch_without_spec_decode()
    raise

Prevention

Do not mix spec-decode and replayssm flags in shared launcher scripts.
Add a config linter that rejects the known-incompatible flag pairs before submission.

When it happens

Trigger: Launching with --use-replayssm plus any speculative-decoding setup that makes num_speculative_tokens > 0 (e.g. a spec-decode config such as ngram, eagle, or medusa with num_speculative_tokens set).

Common situations: A serving config that already had speculative decoding enabled is extended with --use-replayssm; using a shared benchmarking script that always turns on spec decode.

Related errors

AI-assisted analysis of vllm-project/vllm@c794754062 (2026-08-14). Data as JSON: /api/errors/2844f4a584889da7. Report an issue: GitHub.