huggingface/transformers · error · ValueError

assisted generate is only supported for batch_size = 1

Error message

assisted generate is only supported for batch_size = 1

What it means

Error "assisted generate is only supported for batch_size = 1" thrown in huggingface/transformers.

Source

Thrown at src/transformers/generation/utils.py:3670

        # init attention / hidden states / scores tuples
        scores = () if (return_dict_in_generate and output_scores) else None
        raw_logits = () if (return_dict_in_generate and output_logits) else None
        decoder_attentions = () if (return_dict_in_generate and output_attentions) else None
        cross_attentions = () if (return_dict_in_generate and output_attentions) else None
        decoder_hidden_states = () if (return_dict_in_generate and output_hidden_states) else None

        # if model is an encoder-decoder, retrieve encoder attention weights and hidden states
        if return_dict_in_generate and self.config.is_encoder_decoder:
            encoder_attentions = model_kwargs["encoder_outputs"].get("attentions") if output_attentions else None
            encoder_hidden_states = (
                model_kwargs["encoder_outputs"].get("hidden_states") if output_hidden_states else None
            )

        # keep track of which sequences are already finished
        batch_size, cur_len = input_ids.shape[:2]
        if batch_size > 1:
            raise ValueError("assisted generate is only supported for batch_size = 1")
        unfinished_sequences = torch.ones(batch_size, dtype=torch.long, device=input_ids.device)

        this_peer_finished = False
        is_first_iteration = True  # to preserve the same API in the output as other generation methods
        outputs = None
        n_matches = 0
        while self._has_unfinished_sequences(this_peer_finished, synced_gpus, device=input_ids.device):
            cur_len = input_ids.shape[1]

            #  1. Fetch candidate sequences from a `CandidateGenerator` and move to the correct device
            if candidate_generator.requires_model_outputs:
                candidate_input_ids, candidate_logits = candidate_generator.get_candidates(
                    input_ids,
                    model_kwargs=model_kwargs,
                    model_outputs=outputs,
                    is_first_iteration=is_first_iteration,
                    n_last_matches=n_matches,
                )

View on GitHub (pinned to a597f97485)

Solutions

1. Run assisted generation with batch size 1.
2. Loop over batch items individually if you need batched inputs.

When it happens

Trigger: Raised in assisted generation when the batch size is greater than 1.

Common situations: Passing multiple input sequences to generate() together with assistant_model; assisted decoding only supports batch_size=1.

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/0fcb07bf38e0e03b. Report an issue: GitHub.