{"record":{"id":"4d5eb924124591f3","repo":"huggingface/transformers","slug":"assisted-generation-is-not-supported-with-stateful","errorCode":null,"errorMessage":"assisted generation is not supported with stateful models, such as {self.__class__.__name__}","messagePattern":"assisted generation is not supported with stateful models, such as (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/transformers/generation/utils.py","lineNumber":1581,"sourceCode":"                f\"{self.__class__.__name__} only supports {supported_modes}, but got \"\n                f\"generation mode '{generation_mode}'.\"\n            )\n\n        if generation_mode == GenerationMode.BEAM_SEARCH and \"streamer\" in generation_mode_kwargs:\n            raise ValueError(\n                \"`streamer` cannot be used with beam search (yet!). Make sure that `num_beams` is set to 1.\"\n            )\n\n        if generation_mode == GenerationMode.ASSISTED_GENERATION:\n            if generation_config.num_return_sequences > 1:\n                raise ValueError(\n                    \"num_return_sequences has to be 1 when doing assisted generate, \"\n                    f\"but is {generation_config.num_return_sequences}.\"\n                )\n            if self._is_stateful:\n                # In assisted generation we need the ability to confirm whether the model would pick certain tokens,\n                # which is not possible with stateful models (they can't reset to a previous subset of generated text)\n                raise ValueError(\n                    f\"assisted generation is not supported with stateful models, such as {self.__class__.__name__}\"\n                )\n\n        if (\n            assistant_model := generation_mode_kwargs.get(\"assistant_model\")\n        ) is not None and generation_config.speculation_type != \"dflash\":\n            if self.config.is_encoder_decoder and not assistant_model.config.is_encoder_decoder:\n                attributes_to_check = [\"encoder_attention_heads\", \"encoder_ffn_dim\", \"encoder_layers\"]\n                attributes_to_check = [attr for attr in dir(assistant_model.config) if attr in attributes_to_check]\n                are_equal = all(\n                    getattr(self.config, attr) == getattr(assistant_model.config, attr) for attr in attributes_to_check\n                )\n                if not are_equal:\n                    raise ValueError(\n                        \"The main model and the assistant don't have compatible encoder-dependent input shapes. \"\n                        \"Ensure you load the assistant with the correct encoder-decoder class, e.g. `AutoModelForSpeechSeq2Seq` for Whisper.\"\n                    )\n","sourceCodeStart":1563,"sourceCodeEnd":1599,"githubUrl":"https://github.com/huggingface/transformers/blob/a597f974857b3d92939971296bc0deb93d33d780/src/transformers/generation/utils.py#L1563-L1599","documentation":"Assisted generation must re-run the target model on arbitrary candidate tokens, which requires rolling the model state back to a previous position. Models flagged `_is_stateful` keep irreversible state (they cannot reset to an earlier subset of generated text), so speculative verification is impossible and `generate` rejects the combination.","triggerScenarios":"`model.generate(**inputs, assistant_model=assistant)` where the target model class sets `_is_stateful = True` (stateful text-generation models that maintain cross-call internal state), regardless of other parameters.","commonSituations":"Applying speculative decoding to models whose implementation caches state across steps; library upgrades that make a model stateful (or add this validation) so previously-working assisted calls now fail; using generic serving code that always attaches an assistant model.","solutions":["Remove `assistant_model` and generate without assistance for this model.","Use a non-stateful model variant/class for assisted generation.","Check `model._is_stateful` before attaching an assistant in shared pipelines.","If you control the model, ensure the class supports state reset (past-position rollback) before marking it compatible."],"exampleFix":"# before\nout = model.generate(**inputs, assistant_model=assistant)  # ValueError: stateful model\n\n# after\nout = model.generate(**inputs)  # plain generation","handlingStrategy":"validation","validationCode":"if assistant_model is not None and getattr(model, \"_is_stateful\", False):\n    raise ValueError(f\"{type(model).__name__} is stateful; assisted generation unsupported\")","typeGuard":"def supports_assisted(model) -> bool:\n    return not getattr(model, \"_is_stateful\", False)","tryCatchPattern":null,"preventionTips":["Check model._is_stateful before enabling speculative decoding in shared pipelines.","Track transformers release notes: models can gain/lose statefulness flags across versions.","Keep a non-assisted fallback path in inference wrappers."],"tags":["generation","assisted-decoding","stateful-model","incompatible-arguments"],"backgroundTag":null,"analyzedSha":"a597f974857b3d92939971296bc0deb93d33d780","analyzedAt":"2026-08-14T18:24:08.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}