huggingface/transformers · error · ValueError

The main model and the assistant don't have compatible encod

Error message

The main model and the assistant don't have compatible encoder-dependent input shapes. Ensure you load the assistant with the correct encoder-decoder class, e.g. `AutoModelForSpeechSeq2Seq` for Whisper.

What it means

When the main model is encoder-decoder but the assistant is NOT, the assistant must still consume the encoder output; this is only safe when encoder-dependent dimensions match. `generate` compares `encoder_attention_heads`, `encoder_ffn_dim`, `encoder_layers` between the two configs and raises if they differ — the usual cause is loading the assistant with a decoder-only class.

Source

Thrown at src/transformers/generation/utils.py:1595

                )
            if self._is_stateful:
                # In assisted generation we need the ability to confirm whether the model would pick certain tokens,
                # which is not possible with stateful models (they can't reset to a previous subset of generated text)
                raise ValueError(
                    f"assisted generation is not supported with stateful models, such as {self.__class__.__name__}"
                )

        if (
            assistant_model := generation_mode_kwargs.get("assistant_model")
        ) is not None and generation_config.speculation_type != "dflash":
            if self.config.is_encoder_decoder and not assistant_model.config.is_encoder_decoder:
                attributes_to_check = ["encoder_attention_heads", "encoder_ffn_dim", "encoder_layers"]
                attributes_to_check = [attr for attr in dir(assistant_model.config) if attr in attributes_to_check]
                are_equal = all(
                    getattr(self.config, attr) == getattr(assistant_model.config, attr) for attr in attributes_to_check
                )
                if not are_equal:
                    raise ValueError(
                        "The main model and the assistant don't have compatible encoder-dependent input shapes. "
                        "Ensure you load the assistant with the correct encoder-decoder class, e.g. `AutoModelForSpeechSeq2Seq` for Whisper."
                    )

            doc_reference = (
                "(see https://huggingface.co/docs/transformers/en/generation_strategies#universal-assisted-decoding)"
            )
            if self.config.get_text_config().vocab_size == assistant_model.config.get_text_config().vocab_size:
                if "assistant_tokenizer" in generation_mode_kwargs:
                    raise ValueError(
                        f"`assistant_tokenizer` is not required when the main and assistant models use the same tokenizer. Please omit `assistant_tokenizer` from `generate()` {doc_reference}."
                    )
            else:
                if "tokenizer" not in generation_mode_kwargs or "assistant_tokenizer" not in generation_mode_kwargs:
                    raise ValueError(
                        f"The main and assistant models have different tokenizers. Please provide `tokenizer` and `assistant_tokenizer` to `generate()` {doc_reference}."
                    )

View on GitHub (pinned to a597f97485)

Solutions

1. Load the assistant with the encoder-decoder auto class matching the task, e.g. `AutoModelForSpeechSeq2Seq.from_pretrained(distil_model)` for Whisper.
2. Verify the assistant's `encoder_layers`/`encoder_attention_heads`/`encoder_ffn_dim` equal the main model's before calling generate.
3. If dims genuinely differ, pick an assistant whose encoder config matches or skip assisted decoding.
4. Update any cached/pickled assistant model after changing the loading class.

Example fix

# before
assistant = AutoModelForCausalLM.from_pretrained("distil-whisper/distil-small.en")  # wrong class
out = model.generate(**inputs, assistant_model=assistant)  # ValueError: incompatible encoder shapes

# after
from transformers import AutoModelForSpeechSeq2Seq
assistant = AutoModelForSpeechSeq2Seq.from_pretrained("distil-whisper/distil-small.en")
out = model.generate(**inputs, assistant_model=assistant)

if model.config.is_encoder_decoder and not assistant.config.is_encoder_decoder:
    attrs = [a for a in dir(assistant.config) if a in ("encoder_attention_heads", "encoder_ffn_dim", "encoder_layers")]
    assert all(getattr(model.config, a) == getattr(assistant.config, a) for a in attrs), "encoder dims mismatch"

When it happens

Trigger: `model.generate(..., assistant_model=assistant)` where `model.config.is_encoder_decoder=True`, `assistant.config.is_encoder_decoder=False`, and at least one of the encoder attributes present on the assistant differs from the main model — e.g. pairing Whisper with a decoder-only assistant whose dims do not line up.

Common situations: Speeding up speech-seq2seq (Whisper/Seamless) inference with a distilled model loaded via `AutoModelForCausalLM` instead of the matching seq2seq auto class; assistants fine-tuned with a different encoder config; copying assisted-decoding snippets from text-LLM tutorials.

Related errors

• {} is an abstract class. Only classes inheriting this class
• {} is an abstract class. Only classes inheriting this class
• Invalid value for `do_sample`: expected a boolean, got {type
• num_return_sequences has to be 1 when doing assisted generat
• assisted generation is not supported with stateful models, s

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/256888ecfc4d0e24. Report an issue: GitHub.