NousResearch/hermes-agent · error · TimeoutError

Codex auxiliary Responses stream exceeded {float(total_timeo

Error message

Codex auxiliary Responses stream exceeded {float(total_timeout):.1f}s total timeout

What it means

TimeoutError raised by the deadline watchdog inside the Codex auxiliary Responses STREAMING path: the stream exceeded its total timeout budget. On timeout the code also evicts the dead client from the auxiliary client cache (issue #23432) so the next auxiliary call builds a fresh connection instead of reusing a hung one.

Source

Thrown at agent/auxiliary_client.py:1722

                    close()
                except Exception:
                    logger.debug("Codex auxiliary: client close during timeout failed", exc_info=True)
            # The cached auxiliary client wraps this same ``self._client``
            # (or *is* a ``CodexAuxiliaryClient`` whose ``_real_client`` is
            # this instance).  After we close the httpx transport above, the
            # cache must drop that entry — otherwise the next auxiliary call
            # (compression retry, memory flush, etc.) reuses the dead client
            # and fails fast with a connection error.  See issue #23432.
            try:
                _evict_cached_client_instance(self._client)
            except Exception:
                logger.debug("Codex auxiliary: cache eviction on timeout failed", exc_info=True)

        def _check_cancelled() -> None:
            if deadline is not None and time.monotonic() >= deadline:
                if not timed_out.is_set():
                    _close_client_on_timeout()
                raise TimeoutError(_timeout_message())
            try:
                from tools.interrupt import is_interrupted
                # Honor interrupt protection for atomic aux tasks (compression):
                # a mid-flight gateway interrupt must NOT abort the summary call
                # and trigger a degraded fallback marker (#23975). Explicit host
                # cancellation has its own frozen exception; timeouts above still
                # fire and other aux tasks remain interruptible.
                if _aux_interrupt_cancel_requested():
                    raise AuxiliaryExplicitCancellation()
                if is_interrupted() and not _aux_interrupt_protected():
                    raise InterruptedError("Codex auxiliary Responses stream interrupted")
            except (InterruptedError, AuxiliaryExplicitCancellation):
                raise
            except Exception:
                # Interrupt state is a best-effort UX hook; never make it a
                # new failure mode for auxiliary calls.
                pass

View on GitHub (pinned to c896c09c42)

Solutions

Raise the auxiliary task's timeout setting sized to the payload (auxiliary.<task>.timeout)
Retry the operation — the timed-out client is evicted from cache, so the next attempt is fresh
Check network stability/latency to the Codex endpoint
If it recurs only on huge inputs, reduce the auxiliary payload size (e.g. smaller compression chunk)

Example fix

# config.yaml — before
auxiliary:
  compression:
    timeout: 60

# after
auxiliary:
  compression:
    timeout: 300

Defensive patterns

Strategy: retry

Validate before calling

import time

def timeout_budgets_payload(total_timeout: float, estimated_tokens: int, tps: float) -> bool:
    if tps <= 0:
        return False
    return estimated_tokens / tps < total_timeout * 0.8  # 20% headroom

# size the timeout to the work before issuing the streaming call
if not timeout_budgets_payload(cfg.total_timeout, est_tokens, measured_tps):
    cfg.total_timeout = max(cfg.total_timeout, (est_tokens / measured_tps) * 1.5)

Try / catch

for attempt in range(max_retries):
    try:
        return stream_codex_auxiliary(...)
    except TimeoutError as e:
        if "total timeout" in str(e):
            time.sleep(backoff * 2**attempt)  # client was evicted; next try is fresh
            continue
        raise

Prevention

Scale auxiliary timeouts with expected payload size, especially for compression of large contexts
Monitor Codex endpoint latency and raise timeouts before heavy scheduled jobs
Rely on the built-in cache eviction: retrying after a timeout gets a fresh client by design

When it happens

Trigger: A streaming auxiliary call pinned to the openai-codex Responses API whose event stream never completes within total_timeout — hung connection, very large prompt, or slow upstream — until the monotonic deadline passes and _check_cancelled raises.

Common situations: Compression/summary of a near-context-limit conversation under the auxiliary task; flaky network to the Codex endpoint; auxiliary timeout config too low for the payload size.

Understand the failure class

Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

AI-assisted analysis of NousResearch/hermes-agent@c896c09c42 (2026-08-14). Data as JSON: /api/errors/2fd569b27e4f45ce. Report an issue: GitHub.