{"record":{"id":"64dfbff1930e8d74","repo":"huggingface/transformers","slug":"this-model-does-not-support-the-quantized-cache-i","errorCode":null,"errorMessage":"This model does not support the quantized cache. If you want your model to support quantized cache, please open an issue and tag @zucchini-nlp.","messagePattern":"This model does not support the quantized cache\\. If you want your model to support quantized cache, please open an issue and tag @zucchini-nlp\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/transformers/generation/utils.py","lineNumber":2003,"sourceCode":"                    \"and the layer structure will be inferred automatically.\"\n                )\n            # `max_cache_len` lets the static cache be sized for the worst case across calls, so that later calls\n            # with a longer prompt or a larger `max_new_tokens` (up to that ceiling) reuse the same cache instead of\n            # triggering a reallocation (and a `torch.compile` recompilation). Without it, the cache is sized to the\n            # current call's `max_length` only. See #46424.\n            if generation_config.max_cache_len is not None:\n                max_cache_length = max(max_cache_length, generation_config.max_cache_len)\n            cache_batch_size = max(generation_config.num_beams, generation_config.num_return_sequences) * batch_size\n            model_kwargs[cache_name] = self._prepare_static_cache(\n                cache_implementation=generation_config.cache_implementation,\n                batch_size=cache_batch_size,\n                max_cache_len=max_cache_length,\n                prefill_chunk_size=generation_config.prefill_chunk_size,\n                model_kwargs=model_kwargs,\n            )\n        elif generation_config.cache_implementation == \"quantized\":\n            if self.config.is_encoder_decoder or not self._supports_default_dynamic_cache():\n                raise ValueError(\n                    \"This model does not support the quantized cache. If you want your model to support quantized \"\n                    \"cache, please open an issue and tag @zucchini-nlp.\"\n                )\n\n            cache_config = generation_config.cache_config if generation_config.cache_config is not None else {}\n            cache_config.setdefault(\"config\", self.config.get_text_config(decoder=True))\n            backend = cache_config.pop(\"backend\", \"quanto\")\n            model_kwargs[cache_name] = QuantizedCache(backend=backend, **cache_config)\n        # i.e. `cache_implementation` in [None, \"dynamic\", \"offloaded\"]\n        else:\n            model_kwargs[cache_name] = DynamicCache(**dynamic_cache_kwargs)\n\n        if (\n            self.config.is_encoder_decoder\n            and cache_name in model_kwargs\n            and not isinstance(model_kwargs[cache_name], EncoderDecoderCache)\n        ):\n            model_kwargs[cache_name] = EncoderDecoderCache(","sourceCodeStart":1985,"sourceCodeEnd":2021,"githubUrl":"https://github.com/huggingface/transformers/blob/a597f974857b3d92939971296bc0deb93d33d780/src/transformers/generation/utils.py#L1985-L2021","documentation":"`cache_implementation=\"quantized\"` builds a `QuantizedCache`, which is designed for decoder-only models using the standard dynamic-cache path. Encoder-decoder models and models that do not support the default dynamic cache (`_supports_default_dynamic_cache()` false, e.g. mamba/linear-attention-style architectures) cannot use it, so generate raises with a pointer to open a feature request.","triggerScenarios":"`model.generate(**inputs, cache_implementation=\"quantized\")` on an encoder-decoder model (Whisper/T5/...) or on a model whose class does not support the default dynamic cache; also via `generation_config.cache_implementation=\"quantized\"` set in code or a config file.","commonSituations":"Applying the KV-cache quantization recipe from decoder-only docs to a seq2seq model to save memory; a global `cache_implementation=\"quantized\"` default in a serving stack applied to every model including unsupported ones; mamba-family models where the cache format is different.","solutions":["Switch to a supported cache: `cache_implementation=\"dynamic\"` (default) or `\"offloaded\"`, or a `StaticCache` via `\"static\"`.","For encoder-decoder models, quantify the decoder cache length/other levers or use a decoder-only model if quantized KV cache is a hard requirement.","Make per-model cache settings in serving code instead of one global value; gate `\"quantized\"` on `not model.config.is_encoder_decoder and model._supports_default_dynamic_cache()`.","If you need it supported upstream, open the GitHub issue as the message suggests."],"exampleFix":"# before\nout = whisper.generate(**inputs, cache_implementation=\"quantized\")  # encoder-decoder -> ValueError\n\n# after\nout = whisper.generate(**inputs, cache_implementation=\"dynamic\")","handlingStrategy":"validation","validationCode":"if kwargs.get(\"cache_implementation\", model.generation_config.cache_implementation) == \"quantized\":\n    if model.config.is_encoder_decoder or not model._supports_default_dynamic_cache():\n        kwargs[\"cache_implementation\"] = \"dynamic\"  # or raise with a clear message","typeGuard":"def supports_quantized_cache(model) -> bool:\n    return not model.config.is_encoder_decoder and model._supports_default_dynamic_cache()","tryCatchPattern":null,"preventionTips":["Gate 'quantized' on model.config.is_encoder_decoder and _supports_default_dynamic_cache().","Don't apply one global cache_implementation across a heterogeneous model roster.","Budget memory with offloaded/dynamic caches for unsupported architectures."],"tags":["generation","cache","quantized-cache","encoder-decoder","unsupported-feature"],"backgroundTag":null,"analyzedSha":"a597f974857b3d92939971296bc0deb93d33d780","analyzedAt":"2026-08-14T18:24:08.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}