{"record":{"id":"d333a107c10ce5c8","repo":"huggingface/transformers","slug":"low-memory-true-is-not-supported-after-the-beam","errorCode":null,"errorMessage":"`low_memory=True` is not supported after the beam search refactor. Please check the discussion in #35802 *after the PR got merged*, and add a comment there if your questions are not yet answered.","messagePattern":"`low_memory=True` is not supported after the beam search refactor\\. Please check the discussion in #35802 \\*after the PR got merged\\*, and add a comment there if your questions are not yet answered\\.","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/transformers/generation/utils.py","lineNumber":3296,"sourceCode":"        decoder_prompt_len = cur_len\n        this_peer_finished = False\n\n        # At each beam search step, we want to keep top K [K = (number of EOS tokens + 1) * `num_beams`] candidates\n        # with the highest log-probabilities, or sample K continuations without replacement. We gather the top K\n        # (as opposed to `num_beams`, or any number lower than K) so that we have at least `num_beams` sequences\n        # non-finished to continue the live beam search, in case the top `num_beams` all select an EOS token.\n        n_eos_tokens = eos_token_id.shape[0] if eos_token_id is not None else 0\n        beams_to_keep = max(2, 1 + n_eos_tokens) * num_beams\n        top_num_beam_mask = torch.cat(\n            (torch.ones((num_beams), dtype=torch.bool), torch.zeros((beams_to_keep - num_beams), dtype=torch.bool)),\n            dim=0,\n        ).to(input_ids.device)\n\n        # (joao) feature lost in the refactor. Probably won't implement, hurts readability with minimal gains (there\n        # are newer low-memory alternatives like the offloaded cache)\n        sequential = generation_config.low_memory\n        if sequential:\n            raise ValueError(\n                \"`low_memory=True` is not supported after the beam search refactor. Please check the discussion in \"\n                \"#35802 *after the PR got merged*, and add a comment there if your questions are not yet answered.\"\n            )\n\n        # 2. init output tuples\n        all_scores = () if (return_dict_in_generate and output_scores) else None\n        raw_logits = () if (return_dict_in_generate and output_logits) else None\n        beam_indices = () if (return_dict_in_generate and output_logits) else None\n        decoder_attentions = () if (return_dict_in_generate and output_attentions) else None\n        cross_attentions = () if (return_dict_in_generate and output_attentions) else None\n        decoder_hidden_states = () if (return_dict_in_generate and output_hidden_states) else None\n\n        # if model is an encoder-decoder, retrieve encoder attention weights and hidden states\n        if return_dict_in_generate and self.config.is_encoder_decoder:\n            encoder_attentions = model_kwargs[\"encoder_outputs\"].get(\"attentions\") if output_attentions else None\n            encoder_hidden_states = (\n                model_kwargs[\"encoder_outputs\"].get(\"hidden_states\") if output_hidden_states else None\n            )","sourceCodeStart":3278,"sourceCodeEnd":3314,"githubUrl":"https://github.com/huggingface/transformers/blob/a597f974857b3d92939971296bc0deb93d33d780/src/transformers/generation/utils.py#L3278-L3314","documentation":"Error \"`low_memory=True` is not supported after the beam search refactor. Please check the discussion in #35802 *after the PR got merged*, and add a comment there if your questions are not yet answered.\" thrown in huggingface/transformers.","triggerScenarios":"Raised in generate() beam search setup when low_memory=True is requested after the beam search refactor removed support.","commonSituations":"Legacy code passing low_memory=True to generate() with beam search on current transformers versions.","solutions":["Remove `low_memory=True` from the generate call.","Reduce memory via smaller `num_beams` or shorter sequences instead."],"exampleFix":null,"handlingStrategy":null,"validationCode":null,"typeGuard":null,"tryCatchPattern":null,"preventionTips":[],"tags":[],"backgroundTag":null,"analyzedSha":"a597f974857b3d92939971296bc0deb93d33d780","analyzedAt":"2026-08-14T18:24:08.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}