The generated summary did not reduce the conversation size (

Q: What causes "The generated summary did not reduce the conversation size ({before_tokens} tokens before and {after_tokens} tokens after)." in deepset-ai/haystack?

Tighten summary_instruction to demand a terse summary (e.g. 'in at most 200 words, bullet points').

Error message

The generated summary did not reduce the conversation size ({before_tokens} tokens before and {after_tokens} tokens after).

What it means

_apply_summary raises RuntimeError when the summary swap fails to shrink the conversation (haystack/hooks/compaction/summarization.py:452). It re-counts tokens after replacing the summarized messages with the summary and requires after_tokens < before_tokens; otherwise keeping the raw messages is strictly better, so the hook refuses the result.

Source

Thrown at haystack/hooks/compaction/summarization.py:452

        :raises RuntimeError: If the generator returned no usable text, or if the swap did not make the conversation
            smaller, in which case keeping the raw messages is the better outcome.
        """
        replies = result.get("replies") or []
        text = replies[-1].text if replies else None
        if not text or not text.strip():
            raise RuntimeError(
                "The Chat Generator returned no usable text to use as a conversation summary. "
                f"Generator output: {result}."
            )

        summary = ChatMessage.from_user(
            text=f"<conversation_summary>\n{text.strip()}\n</conversation_summary>",
            meta={_COMPACTION_META_KEY: {"strategy": _STRATEGY, "summarized_messages": len(indices)}},
        )
        compacted = _replace_indices(messages=messages, indices=indices, summary=summary)
        after_tokens = token_counter.count(messages=compacted)
        if after_tokens >= before_tokens:
            raise RuntimeError(
                f"The generated summary did not reduce the conversation size ({before_tokens} tokens before and "
                f"{after_tokens} tokens after)."
            )
        return compacted, after_tokens

    def _report_failure(self, error: Exception) -> None:
        """Re-raise a failed summarization or log it, so whatever compacted successfully so far is still returned."""
        if self.raise_on_failure:
            raise error
        logger.warning(
            "Summarizing the conversation for context compaction failed; keeping the last successful result. "
            "Error: {error}",
            error=error,
        )

    def warm_up(self) -> None:
        """Warm up the Chat Generator that writes summaries."""
        if hasattr(self.chat_generator, "warm_up"):

View on GitHub (pinned to e318778c9b)

Solutions

Tighten summary_instruction to demand a terse summary (e.g. 'in at most 200 words, bullet points').
Reduce approximate_summary_tokens / generation max_tokens to cap summary length.
Raise compact_at so more messages get summarized at once, making the swap clearly smaller.
Catch the RuntimeError and fall back to the raw (uncompacted) conversation, as the hook intends.

Example fix

// before
compactor = SummarizationCompactor(gen, summary_instruction="Summarize the conversation.")
// after
compactor = SummarizationCompactor(gen, summary_instruction="Summarize the conversation in at most 150 words of terse bullets.")

Defensive patterns

Strategy: try-catch

Validate before calling

def summary_instruction_is_terse() -> str:
    return "Summarize the conversation in at most 150 words using terse bullet points."
# ensure generation_kwargs cap output too:
generation_kwargs = {"max_tokens": 400}

Try / catch

try:
    compacted = compactor.compact(messages, target_tokens, counter)
except RuntimeError as e:
    if "did not reduce" in str(e):
        logger.warning("summary too large (%s); keeping raw messages", e)
        compacted = messages  # hook semantics: raw messages are the better outcome
    else:
        raise

Prevention

Make summary_instruction explicitly request brevity
Cap summary length via max_tokens
Avoid summarizing very few/short messages where overhead dominates
Raise compact_at so more content is summarized per pass

When it happens

Trigger: The generated summary is as long or longer than the messages it replaces — e.g. a verbose model, an instruction that doesn't ask for brevity, or a tiny number of summarized messages dominated by the <conversation_summary> wrapper overhead.

Common situations: Model ignoring length instructions; summarizing only a few short messages so overhead exceeds savings; summary_instruction prompting for detailed recaps; switching to a model that writes long outputs.

Related errors

AI-assisted analysis of deepset-ai/haystack@e318778c9b (2026-08-30). Data as JSON: /api/errors/173f493039477a1f. Report an issue: GitHub.