huggingface/transformers · error · ValueError

The model must have caching enabled to be performant.

Error message

The model must have caching enabled to be performant.

What it means

Error "The model must have caching enabled to be performant." thrown in huggingface/transformers.

Source

Thrown at src/transformers/integrations/executorch.py:214

        batch_size: int | None = None,
        max_cache_len: int | None = None,
        device: torch.device | None = None,
    ) -> None:
        """
        Initializes the exportable module.

        Args:
            model (`PreTrainedModel`): The pretrained model to wrap.

        Raises:
            ValueError: If the model is configured with a unsupported cache implementation.
        """
        super().__init__()

        config = model.config.get_text_config()

        if not hasattr(config, "use_cache") or config.use_cache is False:
            raise ValueError("The model must have caching enabled to be performant.")

        if hasattr(config, "layer_types") and getattr(config, "sliding_window", None) is not None:
            self.model = TorchExportableModuleWithHybridCache(model, batch_size, max_cache_len, device)
        else:
            # If `layer_types` is not specified explicitly in the config or `sliding_window` is null,
            # there is only 1 type of layers, so export will use `StaticCache` by default.
            logging.info(
                "Using `StaticCache` for export as `layer_types` is not specified or `sliding_window` is `null` in the config."
            )
            self.model = TorchExportableModuleWithStaticCache(model, batch_size, max_cache_len, device)

    def forward(
        self,
        input_ids: torch.Tensor | None = None,
        inputs_embeds: torch.Tensor | None = None,
        cache_position: torch.Tensor | None = None,
    ) -> torch.Tensor:
        """

View on GitHub (pinned to a597f97485)

Solutions

1. Enable caching: set `generation_config.use_cache=True`.

When it happens

Trigger: Raised in ExecuTorch integration when the model runs with caching disabled.

Common situations: Exporting or running a model for ExecuTorch with use_cache=False, which the runtime requires for performance.

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/ea3cdcefb92bd5d4. Report an issue: GitHub.