huggingface/transformers · error · ValueError

`model_outputs` cannot be None, and they need to contain `hi

Error message

`model_outputs` cannot be None, and they need to contain `hidden_states` and `shared_kv_states`

What it means

The Gemma4 shared-KV candidate generator requires the main model's outputs to expose both hidden_states and shared_kv_states (a dict of per-layer (k, v) tuples). If model_outputs is None or lacks either attribute, drafting from the target model is impossible, so it fails fast.

Source

Thrown at src/transformers/generation/candidate_generator.py:1351

        """Generate draft token candidates using the drafter."""
        # This is a trick to skip the first loop of the main model's `_assisted_decoding` method. Since we need the
        # main model's outputs here to get the candidates, we skip the first loop to allow the main model to get the outputs
        # (this is because usually `get_candidates` is called first in the main `_assisted_decoding` loop)
        if is_first_iteration:
            return input_ids, None

        # Early exit if we cannot generate new tokens.
        max_new_tokens = min(int(self.num_assistant_tokens), self.main_model_max_length - input_ids.shape[1] - 1)
        if max_new_tokens <= 0:
            return input_ids, None

        # Make sure we correctly collected all the main model outputs we needed
        if (
            model_outputs is None
            or not hasattr(model_outputs, "hidden_states")
            or not hasattr(model_outputs, "shared_kv_states")
        ):
            raise ValueError(
                "`model_outputs` cannot be None, and they need to contain `hidden_states` and `shared_kv_states`"
            )

        last_hidden_state: torch.Tensor = model_outputs.hidden_states[-1]
        shared_kv_states: dict[str, tuple[torch.Tensor, torch.Tensor]] = model_outputs.shared_kv_states

        # If we validated less tokens, the new `input_ids` are shorter than the last model's outputs, so we need
        # to get the last hidden states and kv states according to the correct length
        current_length = input_ids.shape[1]
        shared_kv_states = {
            k: (v[0][:, :, :current_length, :], v[1][:, :, :current_length, :]) for k, v in shared_kv_states.items()
        }
        # The hidden states have seq_len equal to the last main model's forward pass on all the candidates. We need the
        # last hidden states of only the last validated token
        last_hidden_state = last_hidden_state[:, n_last_matches : n_last_matches + 1]
        last_token_id = input_ids[:, -1:]
        position_ids = torch.tensor([[input_ids.shape[1] - 1]], dtype=torch.long, device=self.assistant_model.device)
        sequence_stopped = torch.zeros(input_ids.shape[0], dtype=torch.bool, device=input_ids.device)

View on GitHub (pinned to a597f97485)

Solutions

1. Run the main model with output_hidden_states=True and ensure it is a Gemma4 model that produces shared_kv_states
2. Use the built-in model.generate assisted-decoding flow, which collects these outputs for you, instead of hand-rolling the loop
3. Skip (return early) when model_outputs is None on iterations where the main model has not run yet

Example fix

# before
outputs = main_model(input_ids)  # hidden states not requested
# after
outputs = main_model(input_ids, output_hidden_states=True)  # shared_kv_states present on Gemma4 targets

def outputs_support_shared_kv(model_outputs) -> bool:
    return (
        model_outputs is not None
        and hasattr(model_outputs, "hidden_states")
        and hasattr(model_outputs, "shared_kv_states")
    )

def has_hidden_and_shared_kv(outputs) -> bool:
    return outputs is not None and getattr(outputs, "hidden_states", None) is not None and getattr(outputs, "shared_kv_states", None) is not None

When it happens

Trigger: Calling this generator's get_candidates with outputs from a main model run with output_hidden_states=False, a model whose forward does not return shared_kv_states (non-Gemma4 target), or None outputs on a non-first iteration.

Common situations: Custom generate loops that call get_candidates manually without requesting hidden states; using a main model that doesn't share KV states with the assistant; partial integration after upgrading transformers.

Related errors

• Expected assistant_model to be a Gemma4AssistantForCausalLM
• Could not find `num_mtp_layers` in the model config. This mo
• `model_outputs` cannot be None, and they need to contain `hi
• `model_outputs` cannot be None and they need to contain `hid
• `assistant_ensemble_weight` must be in the open interval `(0

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/bcca31e8d1088163. Report an issue: GitHub.