zeroclaw-labs/zeroclaw · error · std::io::Error

local IPC endpoint changed while being probed at {}

Error message

local IPC endpoint changed while being probed at {}

What it means

Raised by remove_stale (crates/zeroclaw-runtime/src/rpc/local.rs:496) during the stale-socket teardown handshake: the connect probe got ECONNREFUSED (which normally means 'dead listener'), but re-reading symlink_metadata shows the socket's device/inode identity (SocketIdentity) changed between the first stat and the probe. That means another process unlinked and rebound the endpoint in the window between the two observations, so removing the file now would delete someone else's freshly bound socket — hence ErrorKind::AddrInUse instead of deletion.

Source

Thrown at crates/zeroclaw-runtime/src/rpc/local.rs:496

                    path.display()
                ),
            )
            .into()),
            Err(error) if error.kind() == ErrorKind::NotFound => Ok(()),
            Err(error) if error.kind() != ErrorKind::ConnectionRefused => {
                Err(error).context("probing existing local IPC endpoint")
            }
            Err(_) => {
                let current = match tokio::fs::symlink_metadata(path).await {
                    Ok(metadata) => SocketIdentity::from_metadata(&metadata),
                    Err(error) if error.kind() == ErrorKind::NotFound => return Ok(()),
                    Err(error) => {
                        return Err(error).context("rechecking stale local IPC endpoint");
                    }
                };

                if current != observed {
                    return Err(std::io::Error::new(
                        ErrorKind::AddrInUse,
                        format!(
                            "local IPC endpoint changed while being probed at {}",
                            path.display()
                        ),
                    )
                    .into());
                }

                tokio::fs::remove_file(path)
                    .await
                    .context("removing stale socket")
            }
        }
    }

    pub(super) async fn bind_locked(
        path: &Path,

View on GitHub (pinned to 88bb9c8533)

Solutions

  1. Retry the bind: the race window is tiny, so a bounded retry loop (re-run bind(); each attempt redoes remove_stale with fresh observations) almost always succeeds on the second try.
  2. Eliminate the concurrency: use the default data-dir socket, where EndpointLock serializes binders and this race cannot occur; or add your own single-instance guard (flock/lockfile) around startup on custom paths.
  3. Remove external unbinding: stop scripts/cron jobs that unlink the socket file while daemons may be starting.
  4. If two services legitimately contend, give each its own socket path so they never probe the same endpoint.

Example fix

// before: a single bind call can lose the probe race
let (listener, guard) = rpc::local::bind(&path).await?; // AddrInUse: changed while being probed

// after: bounded retry, since the observation race is transient
let mut last_err = None;
let bound = {
    let mut listener = None;
    for attempt in 0..5 {
        match rpc::local::bind(&path).await {
            Ok(pair) => { listener = Some(pair); break; }
            Err(e) => {
                last_err = Some(e);
                tokio::time::sleep(std::time::Duration::from_millis(50 * (attempt + 1))).await;
            }
        }
    }
    listener
}.ok_or_else(|| last_err.unwrap())?;
Defensive patterns

Strategy: retry

Validate before calling

use tokio::net::UnixStream;

async fn endpoint_quiescent(path: &std::path::Path) -> bool {
    // Best-effort: no live listener AND identity stable across two stats.
    let a = tokio::fs::symlink_metadata(path).await.ok();
    if UnixStream::connect(path).await.is_ok() { return false; }
    let b = tokio::fs::symlink_metadata(path).await.ok();
    matches!((a, b), (Some(x), Some(y)) if x.dev() == y.dev() && x.ino() == y.ino())
        || matches!((a, b), (None, None))
}

Type guard

fn is_probe_race(err: &anyhow::Error) -> bool {
    err.chain()
        .filter_map(|c| c.downcast_ref::<std::io::Error>())
        .any(|e| e.kind() == std::io::ErrorKind::AddrInUse)
        && err.to_string().contains("changed while being probed")
}

Try / catch

let mut delay = std::time::Duration::from_millis(50);
let bound = loop {
    match rpc::local::bind(&path).await {
        Ok(bound) => break bound,
        Err(e) if is_probe_race(&e) && delay < std::time::Duration::from_millis(800) => {
            tokio::time::sleep(delay).await;
            delay *= 2; // transient unlink/rebind race; fresh observations on retry
        }
        Err(e) => return Err(e),
    }
};

Prevention

When it happens

Trigger: A bind() racing with another bind()/restart on the same path: process A stats the socket, process B removes it and binds a new one, A's connect then hits ECONNREFUSED against the old (or new) listener, and A's recheck sees a different (dev, inode) than it first observed. Requires concurrent startups or a restart loop on a shared socket path, typically outside the lifecycle-lock-protected default location.

Common situations: Two daemons auto-restarting simultaneously (systemd Restart=always plus a watchdog, or a supervisor loop) while sharing ZEROCLAW_SOCKET; scripts that 'clean up' /tmp sockets racing with daemon startup; multiple containers mounting the same socket directory; deploy tooling that stops and immediately starts daemons on the same path.

Related errors

AI-assisted analysis of zeroclaw-labs/zeroclaw@88bb9c8533 (2026-08-23). Data as JSON: /api/errors/40e1b16199c6dd38. Report an issue: GitHub.