huggingface/transformers · error · ValueError

assisted generation is not supported with stateful models, s

Error message

assisted generation is not supported with stateful models, such as {self.__class__.__name__}

What it means

Assisted generation must re-run the target model on arbitrary candidate tokens, which requires rolling the model state back to a previous position. Models flagged `_is_stateful` keep irreversible state (they cannot reset to an earlier subset of generated text), so speculative verification is impossible and `generate` rejects the combination.

Source

Thrown at src/transformers/generation/utils.py:1581

                f"{self.__class__.__name__} only supports {supported_modes}, but got "
                f"generation mode '{generation_mode}'."
            )

        if generation_mode == GenerationMode.BEAM_SEARCH and "streamer" in generation_mode_kwargs:
            raise ValueError(
                "`streamer` cannot be used with beam search (yet!). Make sure that `num_beams` is set to 1."
            )

        if generation_mode == GenerationMode.ASSISTED_GENERATION:
            if generation_config.num_return_sequences > 1:
                raise ValueError(
                    "num_return_sequences has to be 1 when doing assisted generate, "
                    f"but is {generation_config.num_return_sequences}."
                )
            if self._is_stateful:
                # In assisted generation we need the ability to confirm whether the model would pick certain tokens,
                # which is not possible with stateful models (they can't reset to a previous subset of generated text)
                raise ValueError(
                    f"assisted generation is not supported with stateful models, such as {self.__class__.__name__}"
                )

        if (
            assistant_model := generation_mode_kwargs.get("assistant_model")
        ) is not None and generation_config.speculation_type != "dflash":
            if self.config.is_encoder_decoder and not assistant_model.config.is_encoder_decoder:
                attributes_to_check = ["encoder_attention_heads", "encoder_ffn_dim", "encoder_layers"]
                attributes_to_check = [attr for attr in dir(assistant_model.config) if attr in attributes_to_check]
                are_equal = all(
                    getattr(self.config, attr) == getattr(assistant_model.config, attr) for attr in attributes_to_check
                )
                if not are_equal:
                    raise ValueError(
                        "The main model and the assistant don't have compatible encoder-dependent input shapes. "
                        "Ensure you load the assistant with the correct encoder-decoder class, e.g. `AutoModelForSpeechSeq2Seq` for Whisper."
                    )

View on GitHub (pinned to a597f97485)

Solutions

Remove `assistant_model` and generate without assistance for this model.
Use a non-stateful model variant/class for assisted generation.
Check `model._is_stateful` before attaching an assistant in shared pipelines.
If you control the model, ensure the class supports state reset (past-position rollback) before marking it compatible.

Example fix

# before
out = model.generate(**inputs, assistant_model=assistant)  # ValueError: stateful model

# after
out = model.generate(**inputs)  # plain generation

Defensive patterns

Strategy: validation

Validate before calling

if assistant_model is not None and getattr(model, "_is_stateful", False):
    raise ValueError(f"{type(model).__name__} is stateful; assisted generation unsupported")

Type guard

def supports_assisted(model) -> bool:
    return not getattr(model, "_is_stateful", False)

Prevention

Check model._is_stateful before enabling speculative decoding in shared pipelines.
Track transformers release notes: models can gain/lose statefulness flags across versions.
Keep a non-assisted fallback path in inference wrappers.

When it happens

Trigger: `model.generate(**inputs, assistant_model=assistant)` where the target model class sets `_is_stateful = True` (stateful text-generation models that maintain cross-call internal state), regardless of other parameters.

Common situations: Applying speculative decoding to models whose implementation caches state across steps; library upgrades that make a model stateful (or add this validation) so previously-working assisted calls now fail; using generic serving code that always attaches an assistant model.

Related errors

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/4d5eb924124591f3. Report an issue: GitHub.