huggingface/transformers · error · ValueError

{self.__class__.__name__} cannot use beam search with a cach

Error message

{self.__class__.__name__} cannot use beam search with a cache currently, as the cache cannot be reordered

What it means

Error "{self.__class__.__name__} cannot use beam search with a cache currently, as the cache cannot be reordered" thrown in huggingface/transformers.

Source

Thrown at src/transformers/generation/utils.py:3487

                decoder_prompt_len=decoder_prompt_len,
                length_penalty=length_penalty,
                early_stopping=early_stopping,
            )

            # g. Prepare remaining data for the next iteration, including computing the stopping condition for
            # beam search as a whole (as opposed to individual beams, i.e. `stopping_criteria`)

            # pluck the cache from the beam indices that will be used in the next iteration
            # NOTE: we need to check if `self._reorder_cache` exists for special models like RAG, RecurrentGemma etc.
            if any(cache_key in model_kwargs for cache_key in ALL_CACHE_NAMES):
                cache_key = next(cache_key for cache_key in ALL_CACHE_NAMES if cache_key in model_kwargs)
                beam_idx = self._flatten_beam_dim(running_beam_indices[..., cur_len - decoder_prompt_len])
                if hasattr(self, "_reorder_cache"):
                    model_kwargs[cache_key] = self._reorder_cache(model_kwargs[cache_key], beam_idx)
                elif hasattr(model_kwargs[cache_key], "reorder_cache"):
                    model_kwargs[cache_key].reorder_cache(beam_idx)
                else:
                    raise ValueError(
                        f"{self.__class__.__name__} cannot use beam search with a cache currently, as the cache cannot be reordered"
                    )

            cur_len = cur_len + 1
            is_early_stop_heuristic_unsatisfied = self._check_early_stop_heuristic(
                is_early_stop_heuristic_unsatisfied=is_early_stop_heuristic_unsatisfied,
                running_beam_scores=running_beam_scores,
                beam_scores=beam_scores,
                is_sent_finished=is_sent_finished,
                cur_len=cur_len,
                max_length=max_length,
                decoder_prompt_len=decoder_prompt_len,
                early_stopping=early_stopping,
                length_penalty=length_penalty,
            )
            this_peer_finished = not self._beam_search_has_unfinished_sequences(
                is_early_stop_heuristic_unsatisfied,
                is_sent_finished,

View on GitHub (pinned to a597f97485)

Solutions

1. Use greedy or sampling decoding instead of beam search for this model.
2. Use a dynamic cache implementation that supports reordering.

When it happens

Trigger: Raised in beam search generation when the model class uses a cache that cannot be reordered for beam hypotheses.

Common situations: Beam search (num_beams>1) on models with non-reorderable caches such as certain linear-attention or Mamba-style models.

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/c5113b672c3431f8. Report an issue: GitHub.