run-llama/llama_index · error · TimeoutError

Response generation timed out after {timeout} seconds

Error message

Response generation timed out after {timeout} seconds

What it means

TimeoutError raised by LangchainChunkBuffer.get_response_gen() in the langchain streaming helpers: the generator polls a token queue every 10ms and, if the complete response has not finished (queue empty and the producer's done-event unset) within `timeout` seconds (default 120), it aborts. It indicates the underlying LLM/streaming callback stalled or is simply slower than the configured budget.

Source

Thrown at llama-index-core/llama_index/core/langchain_helpers/streaming.py:51

        parent_run_id: Optional[UUID] = None,
        tags: Optional[List[str]] = None,
        **kwargs: Any,
    ) -> None:
        self._done.set()

    def get_response_gen(self, timeout: float = 120.0) -> Generator:
        """
        Get response generator with timeout.

        Args:
            timeout (float): Maximum time in seconds to wait for the complete response.
                            Defaults to 120 seconds.

        """
        start_time = time.time()
        while True:
            if time.time() - start_time > timeout:
                raise TimeoutError(
                    f"Response generation timed out after {timeout} seconds"
                )

            if not self._token_queue.empty():
                token = self._token_queue.get_nowait()
                yield token
            elif self._done.is_set():
                break
            else:
                # Small sleep to prevent CPU spinning
                time.sleep(0.01)

View on GitHub (pinned to afd0fef371)

Solutions

1. Raise the budget: get_response_gen(timeout=600.0) sized to your model's worst-case latency.
2. Check the underlying stream/connection — if tokens also stopped arriving, the producer is hung and retrying the request is the real fix.
3. Reduce generation length (max_tokens, tighter prompts) so completion fits the budget.
4. Catch TimeoutError and re-issue the request or degrade gracefully.

Example fix

# before
buffer = LangchainChunkBuffer(...)
for chunk in buffer.get_response_gen():  # default 120s -> TimeoutError on long generations
    ...

# after
for chunk in buffer.get_response_gen(timeout=600.0):
    ...

try:
    for chunk in buffer.get_response_gen(timeout=600.0):
        yield chunk
except TimeoutError:
    # underlying stream stalled: re-issue the request
    yield from retry_request()

When it happens

Trigger: Using the langchain-compatible streaming wrapper around a llama-index query engine and iterating get_response_gen() while the model generates for longer than the timeout; a hung network connection or a callback that never sets the done event; long completions with default timeout=120.

Common situations: Wrapping llama-index query engines in LangChain agents with slow local models; reasoning-heavy prompts on large contexts exceeding 2 minutes; a dropped connection that leaves the producer thread blocked so the done flag is never set.

Understand the failure class

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• Got empty streaming response
• Expected ActionReasoningStep, got {reasoning_step}
• LLM only supports text inputs
• Invalid message content: {message.content!s}
• Streaming is not enabled. Please use chat() instead.

AI-assisted analysis of run-llama/llama_index@afd0fef371 (2026-08-15). Data as JSON: /api/errors/a83f7cb9702aaee9. Report an issue: GitHub.