{"record":{"id":"6af7290573f70314","repo":"huggingface/transformers","slug":"num-return-sequences-has-to-be-1-when-doing-assist","errorCode":null,"errorMessage":"num_return_sequences has to be 1 when doing assisted generate, but is {generation_config.num_return_sequences}.","messagePattern":"num_return_sequences has to be 1 when doing assisted generate, but is (.+?)\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/transformers/generation/utils.py","lineNumber":1574,"sourceCode":"\n    def _validate_generation_mode(\n        self: \"GenerativePreTrainedModel\", generation_mode, generation_config, generation_mode_kwargs\n    ):\n        supported_modes = getattr(self, \"_supported_generation_modes\", None)\n        if supported_modes is not None and generation_mode not in supported_modes:\n            raise ValueError(\n                f\"{self.__class__.__name__} only supports {supported_modes}, but got \"\n                f\"generation mode '{generation_mode}'.\"\n            )\n\n        if generation_mode == GenerationMode.BEAM_SEARCH and \"streamer\" in generation_mode_kwargs:\n            raise ValueError(\n                \"`streamer` cannot be used with beam search (yet!). Make sure that `num_beams` is set to 1.\"\n            )\n\n        if generation_mode == GenerationMode.ASSISTED_GENERATION:\n            if generation_config.num_return_sequences > 1:\n                raise ValueError(\n                    \"num_return_sequences has to be 1 when doing assisted generate, \"\n                    f\"but is {generation_config.num_return_sequences}.\"\n                )\n            if self._is_stateful:\n                # In assisted generation we need the ability to confirm whether the model would pick certain tokens,\n                # which is not possible with stateful models (they can't reset to a previous subset of generated text)\n                raise ValueError(\n                    f\"assisted generation is not supported with stateful models, such as {self.__class__.__name__}\"\n                )\n\n        if (\n            assistant_model := generation_mode_kwargs.get(\"assistant_model\")\n        ) is not None and generation_config.speculation_type != \"dflash\":\n            if self.config.is_encoder_decoder and not assistant_model.config.is_encoder_decoder:\n                attributes_to_check = [\"encoder_attention_heads\", \"encoder_ffn_dim\", \"encoder_layers\"]\n                attributes_to_check = [attr for attr in dir(assistant_model.config) if attr in attributes_to_check]\n                are_equal = all(\n                    getattr(self.config, attr) == getattr(assistant_model.config, attr) for attr in attributes_to_check","sourceCodeStart":1556,"sourceCodeEnd":1592,"githubUrl":"https://github.com/huggingface/transformers/blob/a597f974857b3d92939971296bc0deb93d33d780/src/transformers/generation/utils.py#L1556-L1592","documentation":"Assisted (speculative) generation produces ONE candidate chain that is verified against the target model; it cannot maintain several independent return sequences. When the resolved mode is ASSISTED_GENERATION and `num_return_sequences > 1`, `generate` raises before decoding starts.","triggerScenarios":"`model.generate(**inputs, assistant_model=small_model, num_return_sequences=4)` — also triggered when `num_return_sequences` is inherited from the model's saved `generation_config.json`.","commonSituations":"Porting a diverse-sampling recipe (`num_return_sequences>1`) to a speculative-decoding setup for speed; generation configs copied from a model card that ships `num_return_sequences`; wrappers that add `num_return_sequences` unconditionally.","solutions":["Set `num_return_sequences=1` in the assisted generate call.","If you need multiple sequences, drop `assistant_model` and use plain sampling with `num_return_sequences>1`.","Run assisted generation N times in a loop/batch to get multiple outputs.","Check `model.generation_config.num_return_sequences` and reset it to 1 if a saved config set it."],"exampleFix":"# before\nout = model.generate(**inputs, assistant_model=assistant, num_return_sequences=3)  # ValueError\n\n# after\nout = model.generate(**inputs, assistant_model=assistant, num_return_sequences=1)","handlingStrategy":"validation","validationCode":"if assistant_model is not None:\n    kwargs[\"num_return_sequences\"] = 1","typeGuard":null,"tryCatchPattern":null,"preventionTips":["In assisted-generation code paths, pin num_return_sequences=1 explicitly.","Reset model.generation_config.num_return_sequences to 1 before attaching an assistant.","For multiple outputs, loop assisted generation or use plain sampling without an assistant."],"tags":["generation","assisted-decoding","num-return-sequences","incompatible-arguments"],"backgroundTag":null,"analyzedSha":"a597f974857b3d92939971296bc0deb93d33d780","analyzedAt":"2026-08-14T18:24:08.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}