huggingface/transformers · error · ValueError

`streamer` cannot be used with beam search (yet!). Make sure

Error message

`streamer` cannot be used with beam search (yet!). Make sure that `num_beams` is set to 1.

What it means

Text streaming (`streamer=`) is only implemented for single-path decoding. Beam search maintains multiple candidate sequences per step, which the streamer API cannot represent, so `generate` rejects the combination of `GenerationMode.BEAM_SEARCH` with a `streamer` kwarg.

Source

Thrown at src/transformers/generation/utils.py:1568

        transition_scores = stacked_scores.gather(0, indices)

        # 9. Mask out transition_scores of beams that stopped early
        transition_scores[beam_indices_mask] = 0

        return transition_scores

    def _validate_generation_mode(
        self: "GenerativePreTrainedModel", generation_mode, generation_config, generation_mode_kwargs
    ):
        supported_modes = getattr(self, "_supported_generation_modes", None)
        if supported_modes is not None and generation_mode not in supported_modes:
            raise ValueError(
                f"{self.__class__.__name__} only supports {supported_modes}, but got "
                f"generation mode '{generation_mode}'."
            )

        if generation_mode == GenerationMode.BEAM_SEARCH and "streamer" in generation_mode_kwargs:
            raise ValueError(
                "`streamer` cannot be used with beam search (yet!). Make sure that `num_beams` is set to 1."
            )

        if generation_mode == GenerationMode.ASSISTED_GENERATION:
            if generation_config.num_return_sequences > 1:
                raise ValueError(
                    "num_return_sequences has to be 1 when doing assisted generate, "
                    f"but is {generation_config.num_return_sequences}."
                )
            if self._is_stateful:
                # In assisted generation we need the ability to confirm whether the model would pick certain tokens,
                # which is not possible with stateful models (they can't reset to a previous subset of generated text)
                raise ValueError(
                    f"assisted generation is not supported with stateful models, such as {self.__class__.__name__}"
                )

        if (
            assistant_model := generation_mode_kwargs.get("assistant_model")

View on GitHub (pinned to a597f97485)

Solutions

1. Set `num_beams=1` in the generate call (or remove beam parameters) so decoding stays greedy/sampling while streaming.
2. Keep beam search but drop `streamer` and print the final output instead.
3. If beams came from the model's saved `generation_config.json`, override at call time: `model.generate(..., num_beams=1, streamer=streamer)` or set `model.generation_config.num_beams = 1`.
4. For streaming chat UX, use sampling (`do_sample=True`) which streams token-by-token.

Example fix

# before
streamer = TextIteratorStreamer(tokenizer)
out = model.generate(**inputs, num_beams=5, streamer=streamer)  # ValueError

# after
out = model.generate(**inputs, num_beams=1, do_sample=True, streamer=streamer)

if streamer is not None and kwargs.get("num_beams", getattr(model.generation_config, "num_beams", 1)) > 1:
    kwargs["num_beams"] = 1

When it happens

Trigger: `model.generate(**inputs, streamer=TextStreamer(tokenizer), num_beams=4)` — any parameterization that resolves to beam search mode (e.g. `num_beams>1` without sampling overrides that keep mode greedy/sample) together with a streamer.

Common situations: Adding a `TextIteratorStreamer` for a chat UI onto an existing beam-search config; notebooks that streamed with greedy decoding and later enabled beams for quality; `num_beams` inherited from the model's `generation_config.json` while the code adds a streamer.

Related errors

• `num_return_sequences` ({}) has to be smaller or equal to `n
• TextStreamer only supports batch size 1
• TextDiffusionStreamer only supports batch size 1
• {self.__class__.__name__} only supports {supported_modes}, b
• num_return_sequences has to be 1 when doing assisted generat

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/fb1df5d5bb6eb433. Report an issue: GitHub.