huggingface/transformers · error · RuntimeError

assisted decoding requires a cache

Error message

assisted decoding requires a cache

What it means

Error "assisted decoding requires a cache" thrown in huggingface/transformers.

Source

Thrown at src/transformers/generation/utils.py:3631

            [`~generation.GenerateDecoderOnlyOutput`], [`~generation.GenerateEncoderDecoderOutput`] or
            `torch.LongTensor`: A `torch.LongTensor` containing the generated tokens (default behaviour) or a
            [`~generation.GenerateDecoderOnlyOutput`] if `model.config.is_encoder_decoder=False` and
            `return_dict_in_generate=True` or a [`~generation.GenerateEncoderDecoderOutput`] if
            `model.config.is_encoder_decoder=True`.
        """
        # The cache must be dynamic for assisted generation, and the check must happen AFTER preparing cache
        if not model_kwargs["use_cache"]:
            raise ValueError("assisted generate requires `use_cache=True`")
        if (
            generation_config.cache_implementation in ["static", "hybrid", "sliding_window"]
            or type(model_kwargs.get("past_key_values")) is StaticCache
        ):
            raise ValueError("assisted generate is not supported with Static cache classes`")

        # Make sure we can record past on the cache
        cache = model_kwargs.get("past_key_values")
        if cache is None:
            raise RuntimeError("assisted decoding requires a cache")
        cache.activate_past_recording()

        # Get the candidate generator, given the parameterization
        candidate_generator = self._get_candidate_generator(
            generation_config=generation_config,
            input_ids=input_ids,
            inputs_tensor=inputs_tensor,
            assistant_model=assistant_model,
            logits_processor=logits_processor,
            target_tokenizer=tokenizer,
            assistant_tokenizer=assistant_tokenizer,
            model_kwargs=model_kwargs,
        )
        # init values
        do_sample = generation_config.do_sample
        output_attentions = generation_config.output_attentions
        output_hidden_states = generation_config.output_hidden_states
        output_scores = generation_config.output_scores

View on GitHub (pinned to a597f97485)

Solutions

1. Enable caching (`use_cache=True`) so assisted decoding has a cache to work with.

When it happens

Trigger: Raised in assisted generation when no cache object is available for the candidate generator.

Common situations: Assisted decoding on a model whose forward pass did not return past_key_values / a usable cache.

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/98bda666129646d4. Report an issue: GitHub.