{"record":{"id":"49ccfc880d67cdfd","repo":"NousResearch/hermes-agent","slug":"iron-proxy-did-not-bind-probe-host-tunnel-port","errorCode":null,"errorMessage":"iron-proxy did not bind {probe_host}:{tunnel_port} within {_STARTUP_GRACE_SECONDS}s.  Process was killed.  Last log lines:\n{tail}","messagePattern":"iron-proxy did not bind (.+?):(.+?) within (.+?)s\\.  Process was killed\\.  Last log lines:\n(.+?)","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"agent/proxy_sources/iron_proxy.py","lineNumber":1993,"sourceCode":"            pass\n        raise RuntimeError(\n            f\"iron-proxy exited immediately (code {proc.returncode}). \"\n            f\"Last log lines:\\n{tail}\"\n        )\n\n    # The previous version of this code treated \"process still alive at\n    # deadline\" as success.  That left iron-proxy running but\n    # non-listening on the port, with a pidfile pointing at it —\n    # subsequent restarts would fail with \"address in use\" because the\n    # orphan still held the port.  Require port-listening for success.\n    if not listening:\n        tail = _tail_log(log_path, lines=20)\n        _kill_and_wait(proc, grace_seconds=2)\n        try:\n            pidfile.unlink()\n        except FileNotFoundError:\n            pass\n        raise RuntimeError(\n            f\"iron-proxy did not bind {probe_host}:{tunnel_port} within \"\n            f\"{_STARTUP_GRACE_SECONDS}s.  Process was killed.  \"\n            f\"Last log lines:\\n{tail}\"\n        )\n\n    logger.info(\"Started iron-proxy pid=%s config=%s\", proc.pid, cfg)\n    return get_status()\n\n\ndef _write_pidfile_safely(pidfile: Path, pid: int) -> None:\n    \"\"\"Write ``pid`` to ``pidfile`` with O_EXCL + O_NOFOLLOW + ownership check.\n\n    O_EXCL means \"another start is in progress\" if the file already\n    exists with a live owner — we cleanly fail rather than racing.  When\n    the existing pidfile points at a dead pid (stale crash), we\n    explicitly unlink it before retrying once.\n\n    Side effect: also persists the in-process nonce to disk so","sourceCodeStart":1975,"sourceCodeEnd":2011,"githubUrl":"https://github.com/NousResearch/hermes-agent/blob/c896c09c42910c584c4c7d2325b58c14713ea42c/agent/proxy_sources/iron_proxy.py#L1975-L2011","documentation":"The startup grace window expired with the process still alive but never listening on probe_host:tunnel_port. Because the previous 'alive at deadline = success' behavior left orphans holding the port, the code now kills the process (_kill_and_wait), unlinks the pidfile, and raises with the log tail. Success requires an actual port bind.","triggerScenarios":"start_proxy() where the proxy hangs during init (waiting on a network fetch, DNS, or an unreachable upstream), is configured to bind a different interface/port than probe_host:tunnel_port probes, or a firewall/SELinux blocks the bind.","commonSituations":"probe_host set to 127.0.0.1 while the proxy binds only the LAN interface (or vice versa); tunnel_port mismatch between the probe and the config; DNS/egress hang inside the proxy at init; slow upstream making init exceed the grace window; security policy denying the listen().","solutions":["Verify probe_host/tunnel_port match what the proxy config actually binds (same loopback/interface and port)","Read the log tail — a hung DNS/upstream fetch shows up as the last activity; fix connectivity and retry","If startup legitimately takes longer (cold start, slow network), increase the startup grace allowance rather than retry-looping","If a security module (firewalld/SELinux/AppArmor) blocks the bind, allow the port for the binary"],"exampleFix":"# before: proxy binds 0.0.0.0:8080 but probe checks 127.0.0.1:9090\n# RuntimeError: iron-proxy did not bind 127.0.0.1:9090 within ...s\n# align config.yaml proxy.tunnel_port / probe host with the proxy bind config\nhermes egress start\n\n# after: _port_listening() succeeds within the window","handlingStrategy":"validation","validationCode":"import socket\nfrom pathlib import Path\nimport yaml\n\ndef bind_config_consistent(cfg: Path, probe_host: str, tunnel_port: int) -> bool:\n    conf = yaml.safe_load(cfg.read_text())\n    bind = conf.get('bind') or conf.get('listen') or {}\n    host_ok = str(bind.get('host', '127.0.0.1')) in (probe_host, '0.0.0.0')\n    return host_ok and int(bind.get('port', 0)) == int(tunnel_port)\n\n# require bind_config_consistent(...) and a free port before start_proxy()","typeGuard":null,"tryCatchPattern":"try:\n    start_proxy(...)\nexcept RuntimeError as e:\n    if 'did not bind' in str(e):\n        fix_probe_or_config(str(e))  # align host/port, fix DNS/upstream, raise grace","preventionTips":["Keep the probe host/port and the proxy's bind address/port derived from the same config values","Ensure DNS/egress works before start — a hung init fetch burns the whole grace window","Don't retry in a tight loop; each failed start leaves nothing running (it's killed), so fix the cause first"],"tags":["egress-proxy","network","startup","timeout"],"backgroundTag":null,"analyzedSha":"c896c09c42910c584c4c7d2325b58c14713ea42c","analyzedAt":"2026-08-14T17:18:01.089Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}