huggingface/transformers · error · ValueError

Cannot use prefill chunking without a cache

Error message

Cannot use prefill chunking without a cache

What it means

Error "Cannot use prefill chunking without a cache" thrown in huggingface/transformers.

Source

Thrown at src/transformers/generation/utils.py:3960

            model_inputs = self.prepare_inputs_for_generation(
                input_ids,
                next_sequence_length=next_sequence_length,
                is_first_iteration=is_first_iteration,
                **model_kwargs,
            )
            return self(**model_inputs, return_dict=True)

        # Chunked prefill (for very large contexts)
        else:
            # Even if we are not compiling the forward, flex is always compiled when used. With chunked prefill, we may
            # end up needing just a bit more graphs than the default (which is 8). Doing this avoids very cryptic warnings
            getattr(torch, "_dynamo").config.cache_size_limit = 64

            chunk_size = generation_config.prefill_chunk_size
            input_chunks = torch.split(input_ids, chunk_size, dim=-1)

            if "past_key_values" not in model_kwargs:
                raise ValueError("Cannot use prefill chunking without a cache")

            model_forward = (
                self.get_compiled_call(generation_config.compile_config)
                if self._valid_auto_compile_criteria(model_kwargs, generation_config)
                else self.__call__
            )

            attention_mask = model_kwargs.pop("attention_mask", None)
            position_ids = model_kwargs.pop("position_ids", None)
            past_length = 0
            for input_chunk in input_chunks:
                current_length = past_length + input_chunk.shape[-1]
                if attention_mask is not None:
                    model_kwargs["attention_mask"] = attention_mask[:, :current_length]
                if position_ids is not None:
                    model_kwargs["position_ids"] = position_ids[:, past_length:current_length]
                model_inputs = self.prepare_inputs_for_generation(input_chunk, **model_kwargs)

View on GitHub (pinned to a597f97485)

Solutions

1. Enable `use_cache=True` before using prefill chunking.
2. Disable prefill chunking if caching is unavailable.

When it happens

Trigger: Raised in generate() when prefill chunking is enabled but no cache is in use.

Common situations: Setting a prefill chunk size while use_cache=False or on a model without cache support.

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/02f0ece240160f1e. Report an issue: GitHub.