huggingface/transformers · error · ValueError

This model does not support the quantized cache. If you want

Error message

This model does not support the quantized cache. If you want your model to support quantized cache, please open an issue and tag @zucchini-nlp.

What it means

`cache_implementation="quantized"` builds a `QuantizedCache`, which is designed for decoder-only models using the standard dynamic-cache path. Encoder-decoder models and models that do not support the default dynamic cache (`_supports_default_dynamic_cache()` false, e.g. mamba/linear-attention-style architectures) cannot use it, so generate raises with a pointer to open a feature request.

Source

Thrown at src/transformers/generation/utils.py:2003

                    "and the layer structure will be inferred automatically."
                )
            # `max_cache_len` lets the static cache be sized for the worst case across calls, so that later calls
            # with a longer prompt or a larger `max_new_tokens` (up to that ceiling) reuse the same cache instead of
            # triggering a reallocation (and a `torch.compile` recompilation). Without it, the cache is sized to the
            # current call's `max_length` only. See #46424.
            if generation_config.max_cache_len is not None:
                max_cache_length = max(max_cache_length, generation_config.max_cache_len)
            cache_batch_size = max(generation_config.num_beams, generation_config.num_return_sequences) * batch_size
            model_kwargs[cache_name] = self._prepare_static_cache(
                cache_implementation=generation_config.cache_implementation,
                batch_size=cache_batch_size,
                max_cache_len=max_cache_length,
                prefill_chunk_size=generation_config.prefill_chunk_size,
                model_kwargs=model_kwargs,
            )
        elif generation_config.cache_implementation == "quantized":
            if self.config.is_encoder_decoder or not self._supports_default_dynamic_cache():
                raise ValueError(
                    "This model does not support the quantized cache. If you want your model to support quantized "
                    "cache, please open an issue and tag @zucchini-nlp."
                )

            cache_config = generation_config.cache_config if generation_config.cache_config is not None else {}
            cache_config.setdefault("config", self.config.get_text_config(decoder=True))
            backend = cache_config.pop("backend", "quanto")
            model_kwargs[cache_name] = QuantizedCache(backend=backend, **cache_config)
        # i.e. `cache_implementation` in [None, "dynamic", "offloaded"]
        else:
            model_kwargs[cache_name] = DynamicCache(**dynamic_cache_kwargs)

        if (
            self.config.is_encoder_decoder
            and cache_name in model_kwargs
            and not isinstance(model_kwargs[cache_name], EncoderDecoderCache)
        ):
            model_kwargs[cache_name] = EncoderDecoderCache(

View on GitHub (pinned to a597f97485)

Solutions

1. Switch to a supported cache: `cache_implementation="dynamic"` (default) or `"offloaded"`, or a `StaticCache` via `"static"`.
2. For encoder-decoder models, quantify the decoder cache length/other levers or use a decoder-only model if quantized KV cache is a hard requirement.
3. Make per-model cache settings in serving code instead of one global value; gate `"quantized"` on `not model.config.is_encoder_decoder and model._supports_default_dynamic_cache()`.
4. If you need it supported upstream, open the GitHub issue as the message suggests.

Example fix

# before
out = whisper.generate(**inputs, cache_implementation="quantized")  # encoder-decoder -> ValueError

# after
out = whisper.generate(**inputs, cache_implementation="dynamic")

if kwargs.get("cache_implementation", model.generation_config.cache_implementation) == "quantized":
    if model.config.is_encoder_decoder or not model._supports_default_dynamic_cache():
        kwargs["cache_implementation"] = "dynamic"  # or raise with a clear message

def supports_quantized_cache(model) -> bool:
    return not model.config.is_encoder_decoder and model._supports_default_dynamic_cache()

When it happens

Trigger: `model.generate(**inputs, cache_implementation="quantized")` on an encoder-decoder model (Whisper/T5/...) or on a model whose class does not support the default dynamic cache; also via `generation_config.cache_implementation="quantized"` set in code or a config file.

Common situations: Applying the KV-cache quantization recipe from decoder-only docs to a seq2seq model to save memory; a global `cache_implementation="quantized"` default in a serving stack applied to every model including unsupported ones; mamba-family models where the cache format is different.

Related errors

• `crop` was called, but the current layer does not track past
• Once the sliding window size has been reached, `DynamicSlidi
• `crop` was called, but the current layer does not track past
• Expected {len(combined_cache_data) = } to be 4 or 6. {combin
• One of the two arguments is not a Cache: {type(caches[0]) =

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/64dfbff1930e8d74. Report an issue: GitHub.