huggingface/transformers · error · ValueError

assisted generate requires `use_cache=True`

Error message

assisted generate requires `use_cache=True`

What it means

Error "assisted generate requires `use_cache=True`" thrown in huggingface/transformers.

Source

Thrown at src/transformers/generation/utils.py:3621

                The model used to assist the generation process. If not provided, the main model will be used.
            assistant_tokenizer (`PreTrainedTokenizerBase`, *optional*):
                The tokenizer used for the assistant model. If not provided, the token space is assumed to be the same.
            tokenizer (`PreTrainedTokenizerBase`, *optional*):
                The tokenizer used for the main model. If not provided, the token space is assumed to be the same.
            model_kwargs:
                Additional model specific keyword arguments will be forwarded to the `forward` function of the model.
                If model is an encoder-decoder model the kwargs should include `encoder_outputs`.

        Return:
            [`~generation.GenerateDecoderOnlyOutput`], [`~generation.GenerateEncoderDecoderOutput`] or
            `torch.LongTensor`: A `torch.LongTensor` containing the generated tokens (default behaviour) or a
            [`~generation.GenerateDecoderOnlyOutput`] if `model.config.is_encoder_decoder=False` and
            `return_dict_in_generate=True` or a [`~generation.GenerateEncoderDecoderOutput`] if
            `model.config.is_encoder_decoder=True`.
        """
        # The cache must be dynamic for assisted generation, and the check must happen AFTER preparing cache
        if not model_kwargs["use_cache"]:
            raise ValueError("assisted generate requires `use_cache=True`")
        if (
            generation_config.cache_implementation in ["static", "hybrid", "sliding_window"]
            or type(model_kwargs.get("past_key_values")) is StaticCache
        ):
            raise ValueError("assisted generate is not supported with Static cache classes`")

        # Make sure we can record past on the cache
        cache = model_kwargs.get("past_key_values")
        if cache is None:
            raise RuntimeError("assisted decoding requires a cache")
        cache.activate_past_recording()

        # Get the candidate generator, given the parameterization
        candidate_generator = self._get_candidate_generator(
            generation_config=generation_config,
            input_ids=input_ids,
            inputs_tensor=inputs_tensor,
            assistant_model=assistant_model,

View on GitHub (pinned to a597f97485)

Solutions

1. Set `use_cache=True` in the generation config for assisted generation.

When it happens

Trigger: Raised in assisted generation when use_cache=False in the generation config.

Common situations: Speculative decoding with an assistant model while the model or generation config disables the KV cache.

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/19ff4c9568c38879. Report an issue: GitHub.