NousResearch/hermes-agent · error · TimeoutError

Auxiliary streamed call timed out after {self._total_ceiling

Error message

Auxiliary streamed call timed out after {self._total_ceiling:.0f}s total ceiling (stream still open but over budget)

What it means

A progress-hooked streamed auxiliary call enforces an absolute wall-clock ceiling computed by _aux_stream_total_ceiling(): max(600s, 4x the effective timeout). The accumulator's feed() checks the ceiling on every chunk; a stream that keeps trickling tokens but exceeds the total budget raises TimeoutError. This is the backstop against a degenerate stream that never triggers the idle timeout (one token per idle window forever).

Source

Thrown at agent/auxiliary_client.py:8836

    def __init__(self, model: str = "", total_ceiling: Optional[float] = None):
        self._started = time.monotonic()
        self._total_ceiling = total_ceiling
        self.content_parts: List[str] = []
        self.reasoning_parts: List[str] = []
        self.tool_calls_acc: Dict[int, Dict[str, Any]] = {}
        self.finish_reason = None
        self.usage = None
        self.resp_id = ""
        self.resp_model = model or ""

    def feed(self, chunk: Any) -> None:
        _notify_aux_progress()
        if (
            self._total_ceiling is not None
            and (time.monotonic() - self._started) >= self._total_ceiling
        ):
            raise TimeoutError(
                f"Auxiliary streamed call timed out after {self._total_ceiling:.0f}s "
                "total ceiling (stream still open but over budget)"
            )
        self.resp_id = getattr(chunk, "id", None) or self.resp_id
        self.resp_model = getattr(chunk, "model", None) or self.resp_model
        chunk_usage = getattr(chunk, "usage", None)
        if chunk_usage:
            self.usage = chunk_usage
        choices = getattr(chunk, "choices", None) or []
        if not choices:
            return
        choice = choices[0]
        self.finish_reason = getattr(choice, "finish_reason", None) or self.finish_reason
        delta = getattr(choice, "delta", None)
        if delta is None:
            return
        piece = getattr(delta, "content", None)
        if piece:

View on GitHub (pinned to c896c09c42)

Solutions

Raise the auxiliary task's timeout (config.yaml `auxiliary.<task>.timeout` or the call's timeout arg) — the ceiling scales 4x with it (floor 600s).
Reduce the prompt/context size sent to the auxiliary task.
Switch the auxiliary task to a faster provider/model.
If a proxy is involved, check it is not throttling the stream to a trickle.

Example fix

# config.yaml — raise ceiling from 600s to 1200s for the compression task
# before
auxiliary:
  compression:
    timeout: 150   # ceiling = max(600, 4*150) = 600s
# after
auxiliary:
  compression:
    timeout: 300   # ceiling = max(600, 4*300) = 1200s

Defensive patterns

Strategy: try-catch

Validate before calling

from agent.auxiliary_client import _aux_stream_total_ceiling
ceiling = _aux_stream_total_ceiling(timeout_seconds)
if estimated_generation_time > ceiling:
    raise_system_warning_or_trim_prompt()

Try / catch

try:
    result = await streamed_aux_call(...)
except TimeoutError as e:
    if "total ceiling" in str(e):
        retry_with_smaller_prompt_or_higher_timeout()

Prevention

Size auxiliary prompts so generation comfortably fits max(600s, 4x timeout).
Raise auxiliary.<task>.timeout for slow models rather than streaming huge payloads.
Monitor slow auxiliary streams before they hit the 10-minute floor.

When it happens

Trigger: A streaming auxiliary call (e.g. compression or curator) whose stream stays open longer than max(600s, 4x timeout) — e.g. a 150s timeout gives a 600s floor; a huge prompt with a very slow provider can exceed it.

Common situations: Massive context sent to a slow auxiliary model; a provider that streams keep-alive/comment chunks indefinitely; a misconfigured endpoint that drips empty deltas.

Understand the failure class

Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

AI-assisted analysis of NousResearch/hermes-agent@c896c09c42 (2026-08-14). Data as JSON: /api/errors/ee4aee967c48cef1. Report an issue: GitHub.