{"record":{"id":"bcca31e8d1088163","repo":"huggingface/transformers","slug":"model-outputs-cannot-be-none-and-they-need-to-c","errorCode":null,"errorMessage":"`model_outputs` cannot be None, and they need to contain `hidden_states` and `shared_kv_states`","messagePattern":"`model_outputs` cannot be None, and they need to contain `hidden_states` and `shared_kv_states`","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/transformers/generation/candidate_generator.py","lineNumber":1351,"sourceCode":"        \"\"\"Generate draft token candidates using the drafter.\"\"\"\n        # This is a trick to skip the first loop of the main model's `_assisted_decoding` method. Since we need the\n        # main model's outputs here to get the candidates, we skip the first loop to allow the main model to get the outputs\n        # (this is because usually `get_candidates` is called first in the main `_assisted_decoding` loop)\n        if is_first_iteration:\n            return input_ids, None\n\n        # Early exit if we cannot generate new tokens.\n        max_new_tokens = min(int(self.num_assistant_tokens), self.main_model_max_length - input_ids.shape[1] - 1)\n        if max_new_tokens <= 0:\n            return input_ids, None\n\n        # Make sure we correctly collected all the main model outputs we needed\n        if (\n            model_outputs is None\n            or not hasattr(model_outputs, \"hidden_states\")\n            or not hasattr(model_outputs, \"shared_kv_states\")\n        ):\n            raise ValueError(\n                \"`model_outputs` cannot be None, and they need to contain `hidden_states` and `shared_kv_states`\"\n            )\n\n        last_hidden_state: torch.Tensor = model_outputs.hidden_states[-1]\n        shared_kv_states: dict[str, tuple[torch.Tensor, torch.Tensor]] = model_outputs.shared_kv_states\n\n        # If we validated less tokens, the new `input_ids` are shorter than the last model's outputs, so we need\n        # to get the last hidden states and kv states according to the correct length\n        current_length = input_ids.shape[1]\n        shared_kv_states = {\n            k: (v[0][:, :, :current_length, :], v[1][:, :, :current_length, :]) for k, v in shared_kv_states.items()\n        }\n        # The hidden states have seq_len equal to the last main model's forward pass on all the candidates. We need the\n        # last hidden states of only the last validated token\n        last_hidden_state = last_hidden_state[:, n_last_matches : n_last_matches + 1]\n        last_token_id = input_ids[:, -1:]\n        position_ids = torch.tensor([[input_ids.shape[1] - 1]], dtype=torch.long, device=self.assistant_model.device)\n        sequence_stopped = torch.zeros(input_ids.shape[0], dtype=torch.bool, device=input_ids.device)","sourceCodeStart":1333,"sourceCodeEnd":1369,"githubUrl":"https://github.com/huggingface/transformers/blob/a597f974857b3d92939971296bc0deb93d33d780/src/transformers/generation/candidate_generator.py#L1333-L1369","documentation":"The Gemma4 shared-KV candidate generator requires the main model's outputs to expose both hidden_states and shared_kv_states (a dict of per-layer (k, v) tuples). If model_outputs is None or lacks either attribute, drafting from the target model is impossible, so it fails fast.","triggerScenarios":"Calling this generator's get_candidates with outputs from a main model run with output_hidden_states=False, a model whose forward does not return shared_kv_states (non-Gemma4 target), or None outputs on a non-first iteration.","commonSituations":"Custom generate loops that call get_candidates manually without requesting hidden states; using a main model that doesn't share KV states with the assistant; partial integration after upgrading transformers.","solutions":["Run the main model with output_hidden_states=True and ensure it is a Gemma4 model that produces shared_kv_states","Use the built-in model.generate assisted-decoding flow, which collects these outputs for you, instead of hand-rolling the loop","Skip (return early) when model_outputs is None on iterations where the main model has not run yet"],"exampleFix":"# before\noutputs = main_model(input_ids)  # hidden states not requested\n# after\noutputs = main_model(input_ids, output_hidden_states=True)  # shared_kv_states present on Gemma4 targets","handlingStrategy":"validation","validationCode":"def outputs_support_shared_kv(model_outputs) -> bool:\n    return (\n        model_outputs is not None\n        and hasattr(model_outputs, \"hidden_states\")\n        and hasattr(model_outputs, \"shared_kv_states\")\n    )","typeGuard":"def has_hidden_and_shared_kv(outputs) -> bool:\n    return outputs is not None and getattr(outputs, \"hidden_states\", None) is not None and getattr(outputs, \"shared_kv_states\", None) is not None","tryCatchPattern":null,"preventionTips":["Always pass output_hidden_states=True to the main model in custom speculative-decoding loops","Use built-in generate() which collects required outputs automatically"],"tags":["python","transformers","generation","speculative-decoding","gemma4","hidden-states"],"backgroundTag":null,"analyzedSha":"a597f974857b3d92939971296bc0deb93d33d780","analyzedAt":"2026-08-14T18:24:08.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}