huggingface/transformers · error · ValueError

`model_outputs` cannot be None and they need to contain `hid

Error message

`model_outputs` cannot be None and they need to contain `hidden_states`!

What it means

A block/diffusion-style candidate generator (used in blockwise parallel/diffusion LLM decoding) concatenates hidden states from specific target layers (target_layer_ids) to build candidates. It requires non-None model_outputs exposing hidden_states; otherwise it cannot assemble the cross-layer context.

Source

Thrown at src/transformers/generation/candidate_generator.py:1623

        is_first_iteration: bool,
        n_last_matches: int,
        **kwargs,
    ) -> tuple[torch.LongTensor, torch.FloatTensor | None]:
        """Generate draft token candidates using the drafter."""
        # This is a trick to skip the first loop of the main model's `_assisted_decoding` method. Since we need the
        # main model's outputs here to get the candidates, we skip the first loop to allow the main model to get the outputs
        # (this is because usually `get_candidates` is called first in the main `_assisted_decoding` loop)
        if is_first_iteration:
            return input_ids, None

        # Early exit if we cannot generate new tokens.
        max_new_tokens = min(int(self.block_size), self.main_model_max_length - input_ids.shape[1] - 1)
        if max_new_tokens <= 0:
            return input_ids, None

        # Make sure we correctly collected all the main model outputs we needed
        if model_outputs is None or not hasattr(model_outputs, "hidden_states"):
            raise ValueError("`model_outputs` cannot be None and they need to contain `hidden_states`!")

        num_last_main_model_tokens = n_last_matches + 1 if not self.is_main_model_prefill else input_ids.shape[1] - 1
        # The hidden states hold all tokens from the last main model's forward on all the candidates. We need the
        # hidden states of only accepted tokens thus crop out the rest
        context_hidden_states: torch.Tensor = torch.cat(
            [model_outputs.hidden_states[i + 1][:, :num_last_main_model_tokens] for i in self.target_layer_ids], dim=-1
        )

        # We need to tell the cache how many new states to expect into its k/v states, additional to the "noise" or "diffusion window"
        self.cache.set_previous_accepted_tokens(num_last_main_model_tokens)
        # We need to remvoe the previous "noise" from the cache
        if not self.is_main_model_prefill:
            self.cache.crop(-self.block_size)

        # Here `position_ids`/`attention_mask` are the full sequence inputs, including the last "bonus" token that was just drafted
        # from the main model. We need to slice to get only what the main model just processed. Say the main model just had token
        # positions [2, 3] as input, the tensors contains the data for position [0, 1, 2, 3, 4], i.e. full inputs + new drafted token
        # from last position 3 that was processed

View on GitHub (pinned to a597f97485)

Solutions

1. Always call the main model with output_hidden_states=True before this generator runs
2. Verify the model actually returns per-layer hidden states (output_hidden_states supported and not stripped)
3. Rely on model.generate's built-in path for this generator instead of manual invocation

Example fix

# before
outputs = main_model(input_ids, use_cache=True)
# after
outputs = main_model(input_ids, use_cache=True, output_hidden_states=True)

def outputs_have_hidden_states(model_outputs) -> bool:
    return model_outputs is not None and getattr(model_outputs, "hidden_states", None) is not None

def has_hidden_states(outputs) -> bool:
    hs = getattr(outputs, "hidden_states", None)
    return hs is not None and len(hs) > 0

When it happens

Trigger: Calling this generator after a main-model forward that did not set output_hidden_states=True, or passing model_outputs=None on a non-first iteration of a custom assisted-decoding loop.

Common situations: Diffusion-LLM or blockwise generation integrations where hidden state collection was skipped; custom loops reusing stale/empty ModelOutput objects.

Related errors

• `model_outputs` cannot be None, and they need to contain `hi
• `model_outputs` cannot be None, and they need to contain `hi
• {} is an abstract class. Only classes inheriting this class
• {} is an abstract class. Only classes inheriting this class
• Invalid max_matching_ngram_size or num_output_tokens

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/5d985cd1f40a0af5. Report an issue: GitHub.