huggingface/transformers · error · ValueError

num_return_sequences has to be 1 when doing assisted generat

Error message

num_return_sequences has to be 1 when doing assisted generate, but is {generation_config.num_return_sequences}.

What it means

Assisted (speculative) generation produces ONE candidate chain that is verified against the target model; it cannot maintain several independent return sequences. When the resolved mode is ASSISTED_GENERATION and `num_return_sequences > 1`, `generate` raises before decoding starts.

Source

Thrown at src/transformers/generation/utils.py:1574

    def _validate_generation_mode(
        self: "GenerativePreTrainedModel", generation_mode, generation_config, generation_mode_kwargs
    ):
        supported_modes = getattr(self, "_supported_generation_modes", None)
        if supported_modes is not None and generation_mode not in supported_modes:
            raise ValueError(
                f"{self.__class__.__name__} only supports {supported_modes}, but got "
                f"generation mode '{generation_mode}'."
            )

        if generation_mode == GenerationMode.BEAM_SEARCH and "streamer" in generation_mode_kwargs:
            raise ValueError(
                "`streamer` cannot be used with beam search (yet!). Make sure that `num_beams` is set to 1."
            )

        if generation_mode == GenerationMode.ASSISTED_GENERATION:
            if generation_config.num_return_sequences > 1:
                raise ValueError(
                    "num_return_sequences has to be 1 when doing assisted generate, "
                    f"but is {generation_config.num_return_sequences}."
                )
            if self._is_stateful:
                # In assisted generation we need the ability to confirm whether the model would pick certain tokens,
                # which is not possible with stateful models (they can't reset to a previous subset of generated text)
                raise ValueError(
                    f"assisted generation is not supported with stateful models, such as {self.__class__.__name__}"
                )

        if (
            assistant_model := generation_mode_kwargs.get("assistant_model")
        ) is not None and generation_config.speculation_type != "dflash":
            if self.config.is_encoder_decoder and not assistant_model.config.is_encoder_decoder:
                attributes_to_check = ["encoder_attention_heads", "encoder_ffn_dim", "encoder_layers"]
                attributes_to_check = [attr for attr in dir(assistant_model.config) if attr in attributes_to_check]
                are_equal = all(
                    getattr(self.config, attr) == getattr(assistant_model.config, attr) for attr in attributes_to_check

View on GitHub (pinned to a597f97485)

Solutions

Set `num_return_sequences=1` in the assisted generate call.
If you need multiple sequences, drop `assistant_model` and use plain sampling with `num_return_sequences>1`.
Run assisted generation N times in a loop/batch to get multiple outputs.
Check `model.generation_config.num_return_sequences` and reset it to 1 if a saved config set it.

Example fix

# before
out = model.generate(**inputs, assistant_model=assistant, num_return_sequences=3)  # ValueError

# after
out = model.generate(**inputs, assistant_model=assistant, num_return_sequences=1)

Defensive patterns

Strategy: validation

Validate before calling

if assistant_model is not None:
    kwargs["num_return_sequences"] = 1

Prevention

In assisted-generation code paths, pin num_return_sequences=1 explicitly.
Reset model.generation_config.num_return_sequences to 1 before attaching an assistant.
For multiple outputs, loop assisted generation or use plain sampling without an assistant.

When it happens

Trigger: `model.generate(**inputs, assistant_model=small_model, num_return_sequences=4)` — also triggered when `num_return_sequences` is inherited from the model's saved `generation_config.json`.

Common situations: Porting a diverse-sampling recipe (`num_return_sequences>1`) to a speculative-decoding setup for speed; generation configs copied from a model card that ships `num_return_sequences`; wrappers that add `num_return_sequences` unconditionally.

Related errors

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/6af7290573f70314. Report an issue: GitHub.