huggingface/transformers · error · ValueError

`model_outputs` cannot be None, and they need to contain `hi

Error message

`model_outputs` cannot be None, and they need to contain `hiden_states`

What it means

The MTP candidate generator needs the main model's last hidden state to feed the first MTP layer. On non-first iterations it requires model_outputs to be non-None and expose hidden_states; otherwise drafting cannot proceed. (Note the message's 'hiden_states' is a typo for hidden_states in the source.)

Source

Thrown at src/transformers/generation/candidate_generator.py:1472

    def get_candidates(
        self,
        input_ids: torch.LongTensor,
        model_kwargs: dict[str, Any],
        model_outputs: ModelOutput,
        is_first_iteration: bool,
        n_last_matches: int,
        **kwargs,
    ) -> tuple[torch.LongTensor, torch.FloatTensor | None]:
        # This is a trick to skip the first loop of the main model's `_assisted_decoding` method. Since we need the
        # main model's outputs here to get the candidates, we skip the first loop to allow the main model to get the outputs
        # (this is because usually `get_candidates` is called first in the main `_assisted_decoding` loop)
        if is_first_iteration:
            return input_ids, None

        # Make sure we correctly collected all the main model outputs we needed
        if model_outputs is None or not hasattr(model_outputs, "hidden_states"):
            raise ValueError("`model_outputs` cannot be None, and they need to contain `hiden_states`")

        last_hidden_states: torch.Tensor = model_outputs.hidden_states[-1]

        # Here `input_ids`/`attention_mask`/`position_ids` are the full sequence inputs, including the last token that was
        # just drafted from the main model. We need to slice to get only what the main model just processed, shifted by 1 to the
        # right to take the new token as well. Say the main model just had token positions [2, 3] as input, the tensors
        # contains the data for position [0, 1, 2, 3, 4], i.e. full inputs + new drafted token from last position 3 that was processed.
        # We want to slice to get data for positions [3, 4] for the 1st mtp layer, i.e. same as main model, shifted by 1 to the right.
        # On the other hand, the `full_seq_last_hidden_states` contains data for only already processed positions by the main_model, i.e.
        # one less than `input_ids`/`positions_ids`/`attention_mask`
        num_last_main_model_tokens = n_last_matches + 1 if not self.is_main_model_prefill else input_ids.shape[1] - 1

        # The hidden states have seq_len equal to the last main model's forward pass on all the candidates. We need the
        # last hidden states of only the last validated tokens
        last_hidden_states = last_hidden_states[:, :num_last_main_model_tokens].to(self.device)

        # We need to cache the full last_hidden_states from the main model to be able to correct the mtp cache based on validated tokens
        if self.num_mtp_layers > 1:

View on GitHub (pinned to a597f97485)

Solutions

1. Request hidden states from the main model: output_hidden_states=True on every forward feeding the generator
2. Use the built-in model.generate flow with MTP enabled rather than calling get_candidates by hand
3. Ensure get_candidates is not called with None outputs after the first iteration (check your loop order)

Example fix

# before
outputs = main_model(input_ids, past_key_values=cache)
# after
outputs = main_model(input_ids, past_key_values=cache, output_hidden_states=True)

def outputs_have_hidden_states(model_outputs) -> bool:
    return model_outputs is not None and getattr(model_outputs, "hidden_states", None) is not None

def has_hidden_states(outputs) -> bool:
    return outputs is not None and getattr(outputs, "hidden_states", None) is not None

When it happens

Trigger: Manually calling this generator's get_candidates with model_outputs=None or outputs produced without output_hidden_states=True; or a custom generate loop that skips collecting hidden states before invoking the MTP generator.

Common situations: Custom speculative-decoding loops, integrations that reuse outputs objects from a different code path (e.g. ModelOutput without hidden_states), or disabling hidden state output for performance.

Related errors

• `model_outputs` cannot be None, and they need to contain `hi
• Could not find `num_mtp_layers` in the model config. This mo
• `model_outputs` cannot be None and they need to contain `hid
• {} is an abstract class. Only classes inheriting this class
• {} is an abstract class. Only classes inheriting this class

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/4d957ecee462881e. Report an issue: GitHub.