{"record":{"id":"2fd569b27e4f45ce","repo":"NousResearch/hermes-agent","slug":"codex-auxiliary-responses-stream-exceeded-float-t","errorCode":null,"errorMessage":"Codex auxiliary Responses stream exceeded {float(total_timeout):.1f}s total timeout","messagePattern":"Codex auxiliary Responses stream exceeded (.+?)s total timeout","errorType":"exception","errorClass":"TimeoutError","httpStatus":null,"severity":"error","filePath":"agent/auxiliary_client.py","lineNumber":1722,"sourceCode":"                    close()\n                except Exception:\n                    logger.debug(\"Codex auxiliary: client close during timeout failed\", exc_info=True)\n            # The cached auxiliary client wraps this same ``self._client``\n            # (or *is* a ``CodexAuxiliaryClient`` whose ``_real_client`` is\n            # this instance).  After we close the httpx transport above, the\n            # cache must drop that entry — otherwise the next auxiliary call\n            # (compression retry, memory flush, etc.) reuses the dead client\n            # and fails fast with a connection error.  See issue #23432.\n            try:\n                _evict_cached_client_instance(self._client)\n            except Exception:\n                logger.debug(\"Codex auxiliary: cache eviction on timeout failed\", exc_info=True)\n\n        def _check_cancelled() -> None:\n            if deadline is not None and time.monotonic() >= deadline:\n                if not timed_out.is_set():\n                    _close_client_on_timeout()\n                raise TimeoutError(_timeout_message())\n            try:\n                from tools.interrupt import is_interrupted\n                # Honor interrupt protection for atomic aux tasks (compression):\n                # a mid-flight gateway interrupt must NOT abort the summary call\n                # and trigger a degraded fallback marker (#23975). Explicit host\n                # cancellation has its own frozen exception; timeouts above still\n                # fire and other aux tasks remain interruptible.\n                if _aux_interrupt_cancel_requested():\n                    raise AuxiliaryExplicitCancellation()\n                if is_interrupted() and not _aux_interrupt_protected():\n                    raise InterruptedError(\"Codex auxiliary Responses stream interrupted\")\n            except (InterruptedError, AuxiliaryExplicitCancellation):\n                raise\n            except Exception:\n                # Interrupt state is a best-effort UX hook; never make it a\n                # new failure mode for auxiliary calls.\n                pass\n","sourceCodeStart":1704,"sourceCodeEnd":1740,"githubUrl":"https://github.com/NousResearch/hermes-agent/blob/c896c09c42910c584c4c7d2325b58c14713ea42c/agent/auxiliary_client.py#L1704-L1740","documentation":"TimeoutError raised by the deadline watchdog inside the Codex auxiliary Responses STREAMING path: the stream exceeded its total timeout budget. On timeout the code also evicts the dead client from the auxiliary client cache (issue #23432) so the next auxiliary call builds a fresh connection instead of reusing a hung one.","triggerScenarios":"A streaming auxiliary call pinned to the openai-codex Responses API whose event stream never completes within total_timeout — hung connection, very large prompt, or slow upstream — until the monotonic deadline passes and _check_cancelled raises.","commonSituations":"Compression/summary of a near-context-limit conversation under the auxiliary task; flaky network to the Codex endpoint; auxiliary timeout config too low for the payload size.","solutions":["Raise the auxiliary task's timeout setting sized to the payload (auxiliary.<task>.timeout)","Retry the operation — the timed-out client is evicted from cache, so the next attempt is fresh","Check network stability/latency to the Codex endpoint","If it recurs only on huge inputs, reduce the auxiliary payload size (e.g. smaller compression chunk)"],"exampleFix":"# config.yaml — before\nauxiliary:\n  compression:\n    timeout: 60\n\n# after\nauxiliary:\n  compression:\n    timeout: 300","handlingStrategy":"retry","validationCode":"import time\n\ndef timeout_budgets_payload(total_timeout: float, estimated_tokens: int, tps: float) -> bool:\n    if tps <= 0:\n        return False\n    return estimated_tokens / tps < total_timeout * 0.8  # 20% headroom\n\n# size the timeout to the work before issuing the streaming call\nif not timeout_budgets_payload(cfg.total_timeout, est_tokens, measured_tps):\n    cfg.total_timeout = max(cfg.total_timeout, (est_tokens / measured_tps) * 1.5)","typeGuard":null,"tryCatchPattern":"for attempt in range(max_retries):\n    try:\n        return stream_codex_auxiliary(...)\n    except TimeoutError as e:\n        if \"total timeout\" in str(e):\n            time.sleep(backoff * 2**attempt)  # client was evicted; next try is fresh\n            continue\n        raise","preventionTips":["Scale auxiliary timeouts with expected payload size, especially for compression of large contexts","Monitor Codex endpoint latency and raise timeouts before heavy scheduled jobs","Rely on the built-in cache eviction: retrying after a timeout gets a fresh client by design"],"tags":["timeout","codex","streaming","auxiliary"],"backgroundTag":null,"analyzedSha":"c896c09c42910c584c4c7d2325b58c14713ea42c","analyzedAt":"2026-08-14T17:18:01.089Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}