{"record":{"id":"256888ecfc4d0e24","repo":"huggingface/transformers","slug":"the-main-model-and-the-assistant-don-t-have-compat","errorCode":null,"errorMessage":"The main model and the assistant don't have compatible encoder-dependent input shapes. Ensure you load the assistant with the correct encoder-decoder class, e.g. `AutoModelForSpeechSeq2Seq` for Whisper.","messagePattern":"The main model and the assistant don't have compatible encoder-dependent input shapes\\. Ensure you load the assistant with the correct encoder-decoder class, e\\.g\\. `AutoModelForSpeechSeq2Seq` for Whisper\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/transformers/generation/utils.py","lineNumber":1595,"sourceCode":"                )\n            if self._is_stateful:\n                # In assisted generation we need the ability to confirm whether the model would pick certain tokens,\n                # which is not possible with stateful models (they can't reset to a previous subset of generated text)\n                raise ValueError(\n                    f\"assisted generation is not supported with stateful models, such as {self.__class__.__name__}\"\n                )\n\n        if (\n            assistant_model := generation_mode_kwargs.get(\"assistant_model\")\n        ) is not None and generation_config.speculation_type != \"dflash\":\n            if self.config.is_encoder_decoder and not assistant_model.config.is_encoder_decoder:\n                attributes_to_check = [\"encoder_attention_heads\", \"encoder_ffn_dim\", \"encoder_layers\"]\n                attributes_to_check = [attr for attr in dir(assistant_model.config) if attr in attributes_to_check]\n                are_equal = all(\n                    getattr(self.config, attr) == getattr(assistant_model.config, attr) for attr in attributes_to_check\n                )\n                if not are_equal:\n                    raise ValueError(\n                        \"The main model and the assistant don't have compatible encoder-dependent input shapes. \"\n                        \"Ensure you load the assistant with the correct encoder-decoder class, e.g. `AutoModelForSpeechSeq2Seq` for Whisper.\"\n                    )\n\n            doc_reference = (\n                \"(see https://huggingface.co/docs/transformers/en/generation_strategies#universal-assisted-decoding)\"\n            )\n            if self.config.get_text_config().vocab_size == assistant_model.config.get_text_config().vocab_size:\n                if \"assistant_tokenizer\" in generation_mode_kwargs:\n                    raise ValueError(\n                        f\"`assistant_tokenizer` is not required when the main and assistant models use the same tokenizer. Please omit `assistant_tokenizer` from `generate()` {doc_reference}.\"\n                    )\n            else:\n                if \"tokenizer\" not in generation_mode_kwargs or \"assistant_tokenizer\" not in generation_mode_kwargs:\n                    raise ValueError(\n                        f\"The main and assistant models have different tokenizers. Please provide `tokenizer` and `assistant_tokenizer` to `generate()` {doc_reference}.\"\n                    )\n","sourceCodeStart":1577,"sourceCodeEnd":1613,"githubUrl":"https://github.com/huggingface/transformers/blob/a597f974857b3d92939971296bc0deb93d33d780/src/transformers/generation/utils.py#L1577-L1613","documentation":"When the main model is encoder-decoder but the assistant is NOT, the assistant must still consume the encoder output; this is only safe when encoder-dependent dimensions match. `generate` compares `encoder_attention_heads`, `encoder_ffn_dim`, `encoder_layers` between the two configs and raises if they differ — the usual cause is loading the assistant with a decoder-only class.","triggerScenarios":"`model.generate(..., assistant_model=assistant)` where `model.config.is_encoder_decoder=True`, `assistant.config.is_encoder_decoder=False`, and at least one of the encoder attributes present on the assistant differs from the main model — e.g. pairing Whisper with a decoder-only assistant whose dims do not line up.","commonSituations":"Speeding up speech-seq2seq (Whisper/Seamless) inference with a distilled model loaded via `AutoModelForCausalLM` instead of the matching seq2seq auto class; assistants fine-tuned with a different encoder config; copying assisted-decoding snippets from text-LLM tutorials.","solutions":["Load the assistant with the encoder-decoder auto class matching the task, e.g. `AutoModelForSpeechSeq2Seq.from_pretrained(distil_model)` for Whisper.","Verify the assistant's `encoder_layers`/`encoder_attention_heads`/`encoder_ffn_dim` equal the main model's before calling generate.","If dims genuinely differ, pick an assistant whose encoder config matches or skip assisted decoding.","Update any cached/pickled assistant model after changing the loading class."],"exampleFix":"# before\nassistant = AutoModelForCausalLM.from_pretrained(\"distil-whisper/distil-small.en\")  # wrong class\nout = model.generate(**inputs, assistant_model=assistant)  # ValueError: incompatible encoder shapes\n\n# after\nfrom transformers import AutoModelForSpeechSeq2Seq\nassistant = AutoModelForSpeechSeq2Seq.from_pretrained(\"distil-whisper/distil-small.en\")\nout = model.generate(**inputs, assistant_model=assistant)","handlingStrategy":"validation","validationCode":"if model.config.is_encoder_decoder and not assistant.config.is_encoder_decoder:\n    attrs = [a for a in dir(assistant.config) if a in (\"encoder_attention_heads\", \"encoder_ffn_dim\", \"encoder_layers\")]\n    assert all(getattr(model.config, a) == getattr(assistant.config, a) for a in attrs), \"encoder dims mismatch\"","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Load speech seq2seq assistants with AutoModelForSpeechSeq2Seq (matching auto class per task).","Unit-test assistant/main config compatibility when you add a new assistant checkpoint.","Prefer distilled checkpoints released for the exact main model family."],"tags":["generation","assisted-decoding","encoder-decoder","whisper","model-loading"],"backgroundTag":null,"analyzedSha":"a597f974857b3d92939971296bc0deb93d33d780","analyzedAt":"2026-08-14T18:24:08.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}