{"record":{"id":"5c29d77dd112f750","repo":"huggingface/transformers","slug":"expected-assistant-model-to-be-a-gemma4assistantfo","errorCode":null,"errorMessage":"Expected assistant_model to be a Gemma4AssistantForCausalLM or Gemma4UnifiedAssistantForCausalLM. Got {} This candidate generator requires that the assistant model is able to work from a shared_kv_states dictionary. Currently, only the Gemma4AssistantForCausalLM and Gemma4UnifiedAssistantForCausalLM support this.","messagePattern":"Expected assistant_model to be a Gemma4AssistantForCausalLM or Gemma4UnifiedAssistantForCausalLM\\. Got (.+?) This candidate generator requires that the assistant model is able to work from a shared_kv_states dictionary\\. Currently, only the Gemma4AssistantForCausalLM and Gemma4UnifiedAssistantForCausalLM support this\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/transformers/generation/candidate_generator.py","lineNumber":1294,"sourceCode":"        \"return_shared_kv_states\": True,\n    }\n\n    def __init__(\n        self,\n        input_ids: torch.LongTensor,\n        assistant_model: \"PreTrainedModel\",\n        target_model_input_embeddings: nn.Embedding,\n        generation_config: \"GenerationConfig\",\n        model_kwargs: dict,\n        inputs_tensor: torch.Tensor | None = None,\n        logits_processor: Optional[\"LogitsProcessorList\"] = None,\n        eos_token_id: int | list[int] | torch.Tensor | None = None,\n    ):\n        if (\n            \"Gemma4Assistant\" not in assistant_model.__class__.__name__\n            and \"Gemma4UnifiedAssistant\" not in assistant_model.__class__.__name__\n        ):\n            raise ValueError(\n                f\"Expected assistant_model to be a Gemma4AssistantForCausalLM or Gemma4UnifiedAssistantForCausalLM. Got {assistant_model.__class__.__name__}\"\n                \" This candidate generator requires that the assistant model is able to work from a shared_kv_states\"\n                \" dictionary. Currently, only the Gemma4AssistantForCausalLM and Gemma4UnifiedAssistantForCausalLM support this.\"\n            )\n\n        super().__init__(input_ids, assistant_model, generation_config, model_kwargs, inputs_tensor, logits_processor)\n        self.target_model_input_embeddings = target_model_input_embeddings\n\n        if eos_token_id is None:\n            eos_token_id: set = set()\n\n            if isinstance(self.generation_config.eos_token_id, Iterable):\n                eos_token_id.update(self.generation_config.eos_token_id)\n            elif isinstance(self.generation_config.eos_token_id, int):\n                eos_token_id.add(self.generation_config.eos_token_id)\n\n            if isinstance(self.assistant_generation_config.eos_token_id, Iterable):\n                eos_token_id.update(self.assistant_generation_config.eos_token_id)","sourceCodeStart":1276,"sourceCodeEnd":1312,"githubUrl":"https://github.com/huggingface/transformers/blob/a597f974857b3d92939971296bc0deb93d33d780/src/transformers/generation/candidate_generator.py#L1276-L1312","documentation":"The Gemma4 shared-KV candidate generator (target-model-driven speculative decoding) drafts candidates from the main model's hidden states and shared KV cache, which only Gemma4AssistantForCausalLM / Gemma4UnifiedAssistantForCausalLM support. The constructor rejects any assistant model whose class name contains neither 'Gemma4Assistant' nor 'Gemma4UnifiedAssistant'.","triggerScenarios":"Wiring an ordinary draft model (e.g. a small LlamaForCausalLM or an older Gemma assistant) into the shared-KV candidate path — usually by passing a mismatched assistant_model when the generation config selects the shared-KV generator.","commonSituations":"Mixing model families in assisted decoding, upgrading transformers where Gemma4 introduced this path, or a custom assistant class that does not implement shared_kv_states handling.","solutions":["Use the standard AssistedCandidateGenerator path with any small draft model (pass it as assistant_model without enabling the shared-KV/Gemma4-specific config)","Use Gemma4AssistantForCausalLM or Gemma4UnifiedAssistantForCausalLM as the assistant for this generator","If you wrote a custom assistant that handles shared_kv_states, rename/ensure the class supports the contract instead of relying on the name check"],"exampleFix":"# before\nassistant = LlamaForCausalLM.from_pretrained(\"TinyLlama/TinyLlama-1.1B-Chat-v1.0\")\n# then routed to Gemma4 shared-KV generator -> ValueError\n# after: keep the generic assisted-decoding path\nmodel.generate(inputs, assistant_model=assistant)  # without shared-KV generator selection","handlingStrategy":"validation","validationCode":"def supports_shared_kv_assistant(assistant_model) -> bool:\n    name = assistant_model.__class__.__name__\n    return \"Gemma4Assistant\" in name or \"Gemma4UnifiedAssistant\" in name","typeGuard":"def is_gemma4_assistant(model) -> bool:\n    name = model.__class__.__name__\n    return (\"Gemma4Assistant\" in name) or (\"Gemma4UnifiedAssistant\" in name)","tryCatchPattern":null,"preventionTips":["Match the assistant model family to the candidate generator your generation config selects","Prefer the standard assistant_model= argument so transformers picks the correct generator"],"tags":["python","transformers","generation","speculative-decoding","gemma4","assistant-model"],"backgroundTag":null,"analyzedSha":"a597f974857b3d92939971296bc0deb93d33d780","analyzedAt":"2026-08-14T18:24:08.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}