zylon-ai/private-gpt · error · ValueError

Initial token count exceeds token limit

Error message

Initial token count exceeds token limit

What it means

Raised by TrimmingMemory.aget when initial_token_count already exceeds token_limit. aget subtracts initial_token_count from token_limit to get the remaining budget (max_tokens = token_limit - initial_token_count); a negative budget is nonsensical for either trim strategy, so the method fails fast instead of returning an inverted/negative trim result. initial_token_count typically represents tokens already consumed by the prompt/system parts elsewhere.

Source

Thrown at private_gpt/components/memory/trimming_memory.py:160

            start_on=start_on,
            end_on=end_on,
            tokenizer_fn=tokenizer_fn,
            text_splitter=text_splitter or _default_text_splitter,
            chat_store=chat_store or SimpleChatStore(),
            chat_store_key=chat_store_key,
        )

    async def aget(
        self, input: str | None = None, initial_token_count: int = 0, **kwargs: Any
    ) -> list[ChatMessage]:
        """Get trimmed chat history based on configured strategy."""
        chat_history = await self.aget_all()

        if not chat_history:
            return []

        if initial_token_count > self.token_limit:
            raise ValueError("Initial token count exceeds token limit")

        max_tokens = self.token_limit - initial_token_count

        if self.trim_strategy == TrimStrategy.FIRST:
            return await self._trim_first_max_tokens(chat_history, max_tokens)
        else:
            return await self._trim_last_max_tokens(chat_history, max_tokens)

    async def _trim_first_max_tokens(
        self, messages: list[ChatMessage], max_tokens: int
    ) -> list[ChatMessage]:
        """Keep the first messages up to the token limit."""
        if not messages:
            return messages

        # Find the maximum number of messages we can include
        idx = 0
        for i in range(len(messages)):

View on GitHub (pinned to 4a030776a3)

Solutions

1. Raise token_limit above your worst-case initial_token_count (or derive it from llm.metadata.context_window via from_defaults).
2. Recheck what initial_token_count includes — ensure it is only the fixed, non-history tokens.
3. Shorten the system prompt / static prefix that inflates initial_token_count.
4. Validate initial_token_count <= memory.token_limit before calling aget and degrade gracefully (empty history) if exceeded.

Example fix

# before
history = await memory.aget(initial_token_count=6000)  # token_limit=4096

# after
history = await memory.aget(initial_token_count=6000) if memory.token_limit >= 6000 else []

initial = count_prompt_tokens(system_prompt)  # fixed overhead only
history = await memory.aget(initial_token_count=initial) if initial <= memory.token_limit else []

try:
    history = await memory.aget(initial_token_count=initial)
except ValueError:
    history = []  # no budget left; degrade to empty history

When it happens

Trigger: Calling await memory.aget(input=..., initial_token_count=5000) with token_limit=4096; computing initial_token_count from the current prompt plus system message and exceeding the configured limit; token_limit set smaller than the fixed prompt overhead.

Common situations: Long system prompts eating most of a small token_limit; token_limit derived from a smaller model's context while prompts target a bigger one; counting the same tokens twice (once in initial count, once in history).

Related errors

• Condensed chat history exceeds maximum length after applying
• Unknown memory type: {type}
• Token limit must be set and greater than 0.
• tokenizer_fn must be provided.
• start_on can only be used with 'last' strategy

AI-assisted analysis of zylon-ai/private-gpt@4a030776a3 (2026-08-15). Data as JSON: /api/errors/30edda6a9fef46f8. Report an issue: GitHub.