huggingface/transformers · error · ValueError

assisted generate is not supported with Static cache classes

Error message

assisted generate is not supported with Static cache classes`

What it means

Error "assisted generate is not supported with Static cache classes`" thrown in huggingface/transformers.

Source

Thrown at src/transformers/generation/utils.py:3626

            model_kwargs:
                Additional model specific keyword arguments will be forwarded to the `forward` function of the model.
                If model is an encoder-decoder model the kwargs should include `encoder_outputs`.

        Return:
            [`~generation.GenerateDecoderOnlyOutput`], [`~generation.GenerateEncoderDecoderOutput`] or
            `torch.LongTensor`: A `torch.LongTensor` containing the generated tokens (default behaviour) or a
            [`~generation.GenerateDecoderOnlyOutput`] if `model.config.is_encoder_decoder=False` and
            `return_dict_in_generate=True` or a [`~generation.GenerateEncoderDecoderOutput`] if
            `model.config.is_encoder_decoder=True`.
        """
        # The cache must be dynamic for assisted generation, and the check must happen AFTER preparing cache
        if not model_kwargs["use_cache"]:
            raise ValueError("assisted generate requires `use_cache=True`")
        if (
            generation_config.cache_implementation in ["static", "hybrid", "sliding_window"]
            or type(model_kwargs.get("past_key_values")) is StaticCache
        ):
            raise ValueError("assisted generate is not supported with Static cache classes`")

        # Make sure we can record past on the cache
        cache = model_kwargs.get("past_key_values")
        if cache is None:
            raise RuntimeError("assisted decoding requires a cache")
        cache.activate_past_recording()

        # Get the candidate generator, given the parameterization
        candidate_generator = self._get_candidate_generator(
            generation_config=generation_config,
            input_ids=input_ids,
            inputs_tensor=inputs_tensor,
            assistant_model=assistant_model,
            logits_processor=logits_processor,
            target_tokenizer=tokenizer,
            assistant_tokenizer=assistant_tokenizer,
            model_kwargs=model_kwargs,
        )

View on GitHub (pinned to a597f97485)

Solutions

1. Set `cache_implementation='dynamic'` (or leave default) instead of a Static cache for assisted generation.

When it happens

Trigger: Raised in assisted generation when the cache implementation is a Static cache class.

Common situations: Combining cache_implementation='static' (or StaticCache) with assistant_model speculative decoding.

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/6108d5cbeb7b0b6e. Report an issue: GitHub.