huggingface/transformers · error · ValueError

You passed `inputs_embeds` to `.generate()`, but the model c

Error message

You passed `inputs_embeds` to `.generate()`, but the model class {self.__class__.__name__} doesn't have its forwarding implemented. See the GPT2 implementation for an example (https://github.com/huggingface/transformers/pull/21405), and feel free to open a PR with it!

What it means

ValueError during input preparation: you passed inputs_embeds to .generate() on a decoder-only model whose prepare_inputs_for_generation does not accept an inputs_embeds parameter. The generation loop needs the model to be able to consume embeddings at least for the first forward step; support is opt-in per model architecture (see GPT2's implementation referenced in the message).

Source

Thrown at src/transformers/generation/utils.py:691

            )
        elif inputs_kwarg is not None:
            inputs = inputs_kwarg

        # 3. In the presence of `inputs_embeds` for text models:
        # - decoder-only models should complain if the user attempts to pass `inputs_embeds`, but the model
        # doesn't have its forwarding implemented. `inputs_embeds` is kept in `model_kwargs` and can coexist with
        # input_ids (`inputs_embeds` will be used in the 1st generation step, as opposed to `input_ids`)
        # - encoder-decoder models should complain if the user attempts to pass `inputs_embeds` and `input_ids`, and
        # pull the former to inputs. It will be used in place of `input_ids` to get the encoder hidden states.
        if input_name == "input_ids" and "inputs_embeds" in model_kwargs:
            if model_kwargs["inputs_embeds"] is None:
                model_kwargs.pop("inputs_embeds")
            elif not self.config.is_encoder_decoder:
                has_inputs_embeds_forwarding = "inputs_embeds" in set(
                    inspect.signature(self.prepare_inputs_for_generation).parameters.keys()
                )
                if not has_inputs_embeds_forwarding:
                    raise ValueError(
                        f"You passed `inputs_embeds` to `.generate()`, but the model class {self.__class__.__name__} "
                        "doesn't have its forwarding implemented. See the GPT2 implementation for an example "
                        "(https://github.com/huggingface/transformers/pull/21405), and feel free to open a PR with it!"
                    )
                # In this case, `input_ids` is moved to the `model_kwargs`, so a few automations (like the creation of
                # the attention mask) can rely on the actual model input.
                model_kwargs["input_ids"] = self._maybe_initialize_input_ids_for_generation(
                    inputs, bos_token_id, model_kwargs=model_kwargs
                )
                inputs, input_name = model_kwargs["inputs_embeds"], "inputs_embeds"
            else:
                if inputs is not None:
                    raise ValueError("You passed `inputs_embeds` and `input_ids` to `.generate()`. Please pick one.")
                inputs, input_name = model_kwargs["inputs_embeds"], "inputs_embeds"

        # 4. if `inputs` is still None, try to create `input_ids` from BOS token
        inputs = self._maybe_initialize_input_ids_for_generation(inputs, bos_token_id, model_kwargs)
        return inputs, input_name, model_kwargs

View on GitHub (pinned to a597f97485)

Solutions

1. Switch to a model with inputs_embeds support in generation (GPT2 family, most modern decoder-only models).
2. If it is your custom model, add inputs_embeds to prepare_inputs_for_generation and forward it through (copy the GPT2 pattern from the linked PR #21405).
3. As a workaround, map embeddings back to tokens or append via input_ids instead.

Example fix

# before
out = model.generate(inputs_embeds=soft_prompt_embeds, max_new_tokens=20)  # raises

# after  (custom model)
class MyModel(PreTrainedModel):
    def prepare_inputs_for_generation(self, input_ids, past_key_values=None, inputs_embeds=None, **kwargs):
        ...  # accept and forward inputs_embeds like GPT2 does

import inspect

supports_embeds = "inputs_embeds" in inspect.signature(
    model.prepare_inputs_for_generation
).parameters
if not supports_embeds:
    raise ValueError(f"{type(model).__name__} cannot generate from inputs_embeds")

def model_supports_embeds_generation(model) -> bool:
    return "inputs_embeds" in inspect.signature(model.prepare_inputs_for_generation).parameters

When it happens

Trigger: model.generate(inputs_embeds=embeds, ...) with a model class whose prepare_inputs_for_generation signature lacks inputs_embeds; older custom models or architectures that never implemented embedding-level generation.

Common situations: Prefix-continuation pipelines that precompute hidden states; embedding-based steering/soft-prompt tooling applied to a model without embeds support; upgrading transformers where a custom model's overridden prepare_inputs_for_generation dropped the parameter.

Related errors

• You passed `inputs_embeds` and `input_ids` to `.generate()`.
• Unknown modality for: {model_classname}
• `{pretrained_model_name_or_path}` does not contain a `custom
• `inputs`: {inputs}` were passed alongside {input_name} which
• `bos_token_id` has to be defined when no `input_ids` are pro

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/c20e4a06f72aa0f5. Report an issue: GitHub.