{"record":{"id":"40e1b16199c6dd38","repo":"zeroclaw-labs/zeroclaw","slug":"local-ipc-endpoint-changed-while-being-probed-at","errorCode":null,"errorMessage":"local IPC endpoint changed while being probed at {}","messagePattern":"local IPC endpoint changed while being probed at (.+?)","errorType":"exception","errorClass":"std::io::Error","httpStatus":null,"severity":"error","filePath":"crates/zeroclaw-runtime/src/rpc/local.rs","lineNumber":496,"sourceCode":"                    path.display()\n                ),\n            )\n            .into()),\n            Err(error) if error.kind() == ErrorKind::NotFound => Ok(()),\n            Err(error) if error.kind() != ErrorKind::ConnectionRefused => {\n                Err(error).context(\"probing existing local IPC endpoint\")\n            }\n            Err(_) => {\n                let current = match tokio::fs::symlink_metadata(path).await {\n                    Ok(metadata) => SocketIdentity::from_metadata(&metadata),\n                    Err(error) if error.kind() == ErrorKind::NotFound => return Ok(()),\n                    Err(error) => {\n                        return Err(error).context(\"rechecking stale local IPC endpoint\");\n                    }\n                };\n\n                if current != observed {\n                    return Err(std::io::Error::new(\n                        ErrorKind::AddrInUse,\n                        format!(\n                            \"local IPC endpoint changed while being probed at {}\",\n                            path.display()\n                        ),\n                    )\n                    .into());\n                }\n\n                tokio::fs::remove_file(path)\n                    .await\n                    .context(\"removing stale socket\")\n            }\n        }\n    }\n\n    pub(super) async fn bind_locked(\n        path: &Path,","sourceCodeStart":478,"sourceCodeEnd":514,"githubUrl":"https://github.com/zeroclaw-labs/zeroclaw/blob/88bb9c8533fc57ed7a03e36ca7c9ed2bf8336dcc/crates/zeroclaw-runtime/src/rpc/local.rs#L478-L514","documentation":"Raised by remove_stale (crates/zeroclaw-runtime/src/rpc/local.rs:496) during the stale-socket teardown handshake: the connect probe got ECONNREFUSED (which normally means 'dead listener'), but re-reading symlink_metadata shows the socket's device/inode identity (SocketIdentity) changed between the first stat and the probe. That means another process unlinked and rebound the endpoint in the window between the two observations, so removing the file now would delete someone else's freshly bound socket — hence ErrorKind::AddrInUse instead of deletion.","triggerScenarios":"A bind() racing with another bind()/restart on the same path: process A stats the socket, process B removes it and binds a new one, A's connect then hits ECONNREFUSED against the old (or new) listener, and A's recheck sees a different (dev, inode) than it first observed. Requires concurrent startups or a restart loop on a shared socket path, typically outside the lifecycle-lock-protected default location.","commonSituations":"Two daemons auto-restarting simultaneously (systemd Restart=always plus a watchdog, or a supervisor loop) while sharing ZEROCLAW_SOCKET; scripts that 'clean up' /tmp sockets racing with daemon startup; multiple containers mounting the same socket directory; deploy tooling that stops and immediately starts daemons on the same path.","solutions":["Retry the bind: the race window is tiny, so a bounded retry loop (re-run bind(); each attempt redoes remove_stale with fresh observations) almost always succeeds on the second try.","Eliminate the concurrency: use the default data-dir socket, where EndpointLock serializes binders and this race cannot occur; or add your own single-instance guard (flock/lockfile) around startup on custom paths.","Remove external unbinding: stop scripts/cron jobs that unlink the socket file while daemons may be starting.","If two services legitimately contend, give each its own socket path so they never probe the same endpoint."],"exampleFix":"// before: a single bind call can lose the probe race\nlet (listener, guard) = rpc::local::bind(&path).await?; // AddrInUse: changed while being probed\n\n// after: bounded retry, since the observation race is transient\nlet mut last_err = None;\nlet bound = {\n    let mut listener = None;\n    for attempt in 0..5 {\n        match rpc::local::bind(&path).await {\n            Ok(pair) => { listener = Some(pair); break; }\n            Err(e) => {\n                last_err = Some(e);\n                tokio::time::sleep(std::time::Duration::from_millis(50 * (attempt + 1))).await;\n            }\n        }\n    }\n    listener\n}.ok_or_else(|| last_err.unwrap())?;","handlingStrategy":"retry","validationCode":"use tokio::net::UnixStream;\n\nasync fn endpoint_quiescent(path: &std::path::Path) -> bool {\n    // Best-effort: no live listener AND identity stable across two stats.\n    let a = tokio::fs::symlink_metadata(path).await.ok();\n    if UnixStream::connect(path).await.is_ok() { return false; }\n    let b = tokio::fs::symlink_metadata(path).await.ok();\n    matches!((a, b), (Some(x), Some(y)) if x.dev() == y.dev() && x.ino() == y.ino())\n        || matches!((a, b), (None, None))\n}","typeGuard":"fn is_probe_race(err: &anyhow::Error) -> bool {\n    err.chain()\n        .filter_map(|c| c.downcast_ref::<std::io::Error>())\n        .any(|e| e.kind() == std::io::ErrorKind::AddrInUse)\n        && err.to_string().contains(\"changed while being probed\")\n}","tryCatchPattern":"let mut delay = std::time::Duration::from_millis(50);\nlet bound = loop {\n    match rpc::local::bind(&path).await {\n        Ok(bound) => break bound,\n        Err(e) if is_probe_race(&e) && delay < std::time::Duration::from_millis(800) => {\n            tokio::time::sleep(delay).await;\n            delay *= 2; // transient unlink/rebind race; fresh observations on retry\n        }\n        Err(e) => return Err(e),\n    }\n};","preventionTips":["Use the default data-dir socket so EndpointLock serializes concurrent binders and this race window cannot open.","Never run uncontrolled restart loops against a shared socket path; sequence stop-then-start with a readiness wait.","Keep cleanup scripts (find /tmp -name '*.sock' -delete) away from active daemon socket directories.","Bound your retries: this error signals concurrency on the path; endless retries usually mean two supervisors are fighting and should be surfaced, not hidden."],"tags":["rust","unix-socket","race-condition","daemon","bind"],"backgroundTag":"address-in-use","analyzedSha":"88bb9c8533fc57ed7a03e36ca7c9ed2bf8336dcc","analyzedAt":"2026-08-23T01:07:41.857Z","schemaVersion":2},"datasetVersion":"2026-08-23T08:06:27.607Z"}