huggingface/transformers · error · ValueError

If `is_encoder_decoder` is True, make sure that `encoder_out

Error message

If `is_encoder_decoder` is True, make sure that `encoder_outputs` is defined.

What it means

ValueError from _expand_inputs_for_generation: for encoder-decoder models, generation with num_return_sequences > 1 or beam search expands inputs and model_kwargs by a factor; the encoder_outputs must already be present in model_kwargs at that point. If it is None/missing, decoding has nothing to condition on, so the expansion helper refuses.

Source

Thrown at src/transformers/generation/utils.py:935

        # Do not call torch.repeat_interleave if expand_size is 1 because it clones
        # the input tensor and thus requires more memory although no change is applied
        if expand_size == 1:
            return input_ids, model_kwargs

        def _expand_dict_for_generation(dict_to_expand):
            for key in dict_to_expand:
                if dict_to_expand[key] is not None and isinstance(dict_to_expand[key], torch.Tensor):
                    dict_to_expand[key] = dict_to_expand[key].repeat_interleave(expand_size, dim=0)
            return dict_to_expand

        if input_ids is not None:
            input_ids = input_ids.repeat_interleave(expand_size, dim=0)

        model_kwargs = _expand_dict_for_generation(model_kwargs)

        if is_encoder_decoder:
            if model_kwargs.get("encoder_outputs") is None:
                raise ValueError("If `is_encoder_decoder` is True, make sure that `encoder_outputs` is defined.")
            model_kwargs["encoder_outputs"] = _expand_dict_for_generation(model_kwargs["encoder_outputs"])

        return input_ids, model_kwargs

    def _update_model_kwargs_for_generation(
        self,
        outputs: ModelOutput,
        model_kwargs: dict[str, Any],
        is_encoder_decoder: bool = False,
        num_new_tokens: int = 1,
    ) -> dict[str, Any]:
        """
        Update the model kwargs to account for the `num_new_tokens` new tokens that were just generated.
        That is, update the `attention_mask`, `position_ids`, and `token_type_ids` to account for the
        new tokens of the total sequence.
        Note that this function never slices inputs, this is performed in `prepare_inputs_for_generation`.
        """
        # update past_key_values keeping its naming used in model code

View on GitHub (pinned to a597f97485)

Solutions

1. Run the encoder first and put its output in model_kwargs: model_kwargs['encoder_outputs'] = model.get_encoder()(...), then call generate.
2. Or simply call model.generate(input_ids=...) on the full encoder-decoder model so the encoder runs internally.
3. If you intentionally precompute encoder outputs, make sure they survive in model_kwargs until expansion.

Example fix

# before
model._expand_inputs_for_generation(input_ids, model_kwargs, expand_size=4, is_encoder_decoder=True)
# model_kwargs lacks encoder_outputs

# after
model_kwargs['encoder_outputs'] = model.get_encoder()(encoder_input_ids)
input_ids, model_kwargs = model._expand_inputs_for_generation(input_ids, model_kwargs, expand_size=4, is_encoder_decoder=True)

if is_encoder_decoder and model_kwargs.get("encoder_outputs") is None:
    model_kwargs["encoder_outputs"] = model.get_encoder()(input_ids=encoder_ids)

When it happens

Trigger: Calling low-level generation utilities (or a custom loop) with is_encoder_decoder=True but no encoder_outputs in model_kwargs — e.g. calling model.generate() on an encoder-decoder model after popping encoder_outputs, or using decoder_start_token_ids without running the encoder first.

Common situations: Custom beam-search reimplementations that reuse _expand_inputs_for_generation; caches cleared between encoder and decoder phases; multi-modal encoder-decoder setups where the encoder step was skipped.

Related errors

• You passed `inputs_embeds` and `input_ids` to `.generate()`.
• `decoder_start_token_id` expected to have length {batch_size
• Expected {len(combined_cache_data) = } to be 4 or 6. {combin
• One of the two arguments is not a Cache: {type(caches[0]) =
• Expected 1 or 2 arguments, got {len(caches)}

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/936696e9be82e97b. Report an issue: GitHub.