{"record":{"id":"ee4aee967c48cef1","repo":"NousResearch/hermes-agent","slug":"auxiliary-streamed-call-timed-out-after-self-tot","errorCode":null,"errorMessage":"Auxiliary streamed call timed out after {self._total_ceiling:.0f}s total ceiling (stream still open but over budget)","messagePattern":"Auxiliary streamed call timed out after (.+?)s total ceiling \\(stream still open but over budget\\)","errorType":"exception","errorClass":"TimeoutError","httpStatus":null,"severity":"error","filePath":"agent/auxiliary_client.py","lineNumber":8836,"sourceCode":"\n    def __init__(self, model: str = \"\", total_ceiling: Optional[float] = None):\n        self._started = time.monotonic()\n        self._total_ceiling = total_ceiling\n        self.content_parts: List[str] = []\n        self.reasoning_parts: List[str] = []\n        self.tool_calls_acc: Dict[int, Dict[str, Any]] = {}\n        self.finish_reason = None\n        self.usage = None\n        self.resp_id = \"\"\n        self.resp_model = model or \"\"\n\n    def feed(self, chunk: Any) -> None:\n        _notify_aux_progress()\n        if (\n            self._total_ceiling is not None\n            and (time.monotonic() - self._started) >= self._total_ceiling\n        ):\n            raise TimeoutError(\n                f\"Auxiliary streamed call timed out after {self._total_ceiling:.0f}s \"\n                \"total ceiling (stream still open but over budget)\"\n            )\n        self.resp_id = getattr(chunk, \"id\", None) or self.resp_id\n        self.resp_model = getattr(chunk, \"model\", None) or self.resp_model\n        chunk_usage = getattr(chunk, \"usage\", None)\n        if chunk_usage:\n            self.usage = chunk_usage\n        choices = getattr(chunk, \"choices\", None) or []\n        if not choices:\n            return\n        choice = choices[0]\n        self.finish_reason = getattr(choice, \"finish_reason\", None) or self.finish_reason\n        delta = getattr(choice, \"delta\", None)\n        if delta is None:\n            return\n        piece = getattr(delta, \"content\", None)\n        if piece:","sourceCodeStart":8818,"sourceCodeEnd":8854,"githubUrl":"https://github.com/NousResearch/hermes-agent/blob/c896c09c42910c584c4c7d2325b58c14713ea42c/agent/auxiliary_client.py#L8818-L8854","documentation":"A progress-hooked streamed auxiliary call enforces an absolute wall-clock ceiling computed by _aux_stream_total_ceiling(): max(600s, 4x the effective timeout). The accumulator's feed() checks the ceiling on every chunk; a stream that keeps trickling tokens but exceeds the total budget raises TimeoutError. This is the backstop against a degenerate stream that never triggers the idle timeout (one token per idle window forever).","triggerScenarios":"A streaming auxiliary call (e.g. compression or curator) whose stream stays open longer than max(600s, 4x timeout) — e.g. a 150s timeout gives a 600s floor; a huge prompt with a very slow provider can exceed it.","commonSituations":"Massive context sent to a slow auxiliary model; a provider that streams keep-alive/comment chunks indefinitely; a misconfigured endpoint that drips empty deltas.","solutions":["Raise the auxiliary task's timeout (config.yaml `auxiliary.<task>.timeout` or the call's timeout arg) — the ceiling scales 4x with it (floor 600s).","Reduce the prompt/context size sent to the auxiliary task.","Switch the auxiliary task to a faster provider/model.","If a proxy is involved, check it is not throttling the stream to a trickle."],"exampleFix":"# config.yaml — raise ceiling from 600s to 1200s for the compression task\n# before\nauxiliary:\n  compression:\n    timeout: 150   # ceiling = max(600, 4*150) = 600s\n# after\nauxiliary:\n  compression:\n    timeout: 300   # ceiling = max(600, 4*300) = 1200s","handlingStrategy":"try-catch","validationCode":"from agent.auxiliary_client import _aux_stream_total_ceiling\nceiling = _aux_stream_total_ceiling(timeout_seconds)\nif estimated_generation_time > ceiling:\n    raise_system_warning_or_trim_prompt()","typeGuard":null,"tryCatchPattern":"try:\n    result = await streamed_aux_call(...)\nexcept TimeoutError as e:\n    if \"total ceiling\" in str(e):\n        retry_with_smaller_prompt_or_higher_timeout()","preventionTips":["Size auxiliary prompts so generation comfortably fits max(600s, 4x timeout).","Raise auxiliary.<task>.timeout for slow models rather than streaming huge payloads.","Monitor slow auxiliary streams before they hit the 10-minute floor."],"tags":["auxiliary","streaming","timeout","budget"],"backgroundTag":null,"analyzedSha":"c896c09c42910c584c4c7d2325b58c14713ea42c","analyzedAt":"2026-08-14T17:18:01.089Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}