NousResearch/hermes-agent · error · RuntimeError

iron-proxy did not bind {probe_host}:{tunnel_port} within {_

Error message

iron-proxy did not bind {probe_host}:{tunnel_port} within {_STARTUP_GRACE_SECONDS}s.  Process was killed.  Last log lines:
{tail}

What it means

The startup grace window expired with the process still alive but never listening on probe_host:tunnel_port. Because the previous 'alive at deadline = success' behavior left orphans holding the port, the code now kills the process (_kill_and_wait), unlinks the pidfile, and raises with the log tail. Success requires an actual port bind.

Source

Thrown at agent/proxy_sources/iron_proxy.py:1993

            pass
        raise RuntimeError(
            f"iron-proxy exited immediately (code {proc.returncode}). "
            f"Last log lines:\n{tail}"
        )

    # The previous version of this code treated "process still alive at
    # deadline" as success.  That left iron-proxy running but
    # non-listening on the port, with a pidfile pointing at it —
    # subsequent restarts would fail with "address in use" because the
    # orphan still held the port.  Require port-listening for success.
    if not listening:
        tail = _tail_log(log_path, lines=20)
        _kill_and_wait(proc, grace_seconds=2)
        try:
            pidfile.unlink()
        except FileNotFoundError:
            pass
        raise RuntimeError(
            f"iron-proxy did not bind {probe_host}:{tunnel_port} within "
            f"{_STARTUP_GRACE_SECONDS}s.  Process was killed.  "
            f"Last log lines:\n{tail}"
        )

    logger.info("Started iron-proxy pid=%s config=%s", proc.pid, cfg)
    return get_status()


def _write_pidfile_safely(pidfile: Path, pid: int) -> None:
    """Write ``pid`` to ``pidfile`` with O_EXCL + O_NOFOLLOW + ownership check.

    O_EXCL means "another start is in progress" if the file already
    exists with a live owner — we cleanly fail rather than racing.  When
    the existing pidfile points at a dead pid (stale crash), we
    explicitly unlink it before retrying once.

    Side effect: also persists the in-process nonce to disk so

View on GitHub (pinned to c896c09c42)

Solutions

Verify probe_host/tunnel_port match what the proxy config actually binds (same loopback/interface and port)
Read the log tail — a hung DNS/upstream fetch shows up as the last activity; fix connectivity and retry
If startup legitimately takes longer (cold start, slow network), increase the startup grace allowance rather than retry-looping
If a security module (firewalld/SELinux/AppArmor) blocks the bind, allow the port for the binary

Example fix

# before: proxy binds 0.0.0.0:8080 but probe checks 127.0.0.1:9090
# RuntimeError: iron-proxy did not bind 127.0.0.1:9090 within ...s
# align config.yaml proxy.tunnel_port / probe host with the proxy bind config
hermes egress start

# after: _port_listening() succeeds within the window

Defensive patterns

Strategy: validation

Validate before calling

import socket
from pathlib import Path
import yaml

def bind_config_consistent(cfg: Path, probe_host: str, tunnel_port: int) -> bool:
    conf = yaml.safe_load(cfg.read_text())
    bind = conf.get('bind') or conf.get('listen') or {}
    host_ok = str(bind.get('host', '127.0.0.1')) in (probe_host, '0.0.0.0')
    return host_ok and int(bind.get('port', 0)) == int(tunnel_port)

# require bind_config_consistent(...) and a free port before start_proxy()

Try / catch

try:
    start_proxy(...)
except RuntimeError as e:
    if 'did not bind' in str(e):
        fix_probe_or_config(str(e))  # align host/port, fix DNS/upstream, raise grace

Prevention

Keep the probe host/port and the proxy's bind address/port derived from the same config values
Ensure DNS/egress works before start — a hung init fetch burns the whole grace window
Don't retry in a tight loop; each failed start leaves nothing running (it's killed), so fix the cause first

When it happens

Trigger: start_proxy() where the proxy hangs during init (waiting on a network fetch, DNS, or an unreachable upstream), is configured to bind a different interface/port than probe_host:tunnel_port probes, or a firewall/SELinux blocks the bind.

Common situations: probe_host set to 127.0.0.1 while the proxy binds only the LAN interface (or vice versa); tunnel_port mismatch between the probe and the config; DNS/egress hang inside the proxy at init; slow upstream making init exceed the grace window; security policy denying the listen().

Related errors

AI-assisted analysis of NousResearch/hermes-agent@c896c09c42 (2026-08-14). Data as JSON: /api/errors/49ccfc880d67cdfd. Report an issue: GitHub.