xai-org/grok-build · error

connect timed out

Error message

connect timed out

What it means

In `connect_unix` of the NFS client, after `connect(2)` on a Unix domain socket returns EINPROGRESS, libc::poll waits up to `timeout` for the socket to become writable. A poll result of 0 means the timeout elapsed without the connection completing, so a TimedOut io::Error labeled 'connect timed out' is raised with a 'connect' context. It indicates the NFS daemon's socket never accepted the connection.

Source

Thrown at crates/codegen/xai-fast-worktree/src/nfs/client.rs:632

            std::ptr::addr_of!(addr).cast::<libc::sockaddr>(),
            addr_len,
        )
    };
    if rc != 0 {
        let err = std::io::Error::last_os_error();
        if err.raw_os_error() != Some(libc::EINPROGRESS) {
            return Err(err).context("connect");
        }
        let mut pfd = libc::pollfd {
            fd: raw,
            events: libc::POLLOUT,
            revents: 0,
        };
        let ms = i32::try_from(timeout.as_millis()).unwrap_or(i32::MAX);
        // SAFETY: `pfd` is one pollfd we own for the duration of the call.
        let pr = unsafe { libc::poll(std::ptr::addr_of_mut!(pfd), 1, ms) };
        if pr == 0 {
            return Err(std::io::Error::new(
                std::io::ErrorKind::TimedOut,
                "connect timed out",
            ))
            .context("connect");
        }
        if pr < 0 {
            return Err(std::io::Error::last_os_error()).context("poll");
        }
        let mut so_err: libc::c_int = 0;
        let mut len = std::mem::size_of::<libc::c_int>() as libc::socklen_t;
        // SAFETY: `so_err`/`len` are valid stack integers; `raw` is our socket.
        let gs = unsafe {
            libc::getsockopt(
                raw,
                libc::SOL_SOCKET,
                libc::SO_ERROR,
                std::ptr::addr_of_mut!(so_err).cast(),
                std::ptr::addr_of_mut!(len),

View on GitHub (pinned to bc7f02eddd)

Solutions

1. Check the daemon is alive and listening on the socket path (ls -l the socket, check daemon process/logs).
2. Remove a stale socket file and restart the daemon so a fresh one is bound.
3. Increase the connect timeout to tolerate loaded systems.
4. Fall back to the non-NFS (copy) worktree strategy on TimedOut, mirroring the StorageFull fallback path.

Example fix

// before
let resp = nfs_client.call(req).await?;
// after
let resp = nfs_client.call(req).await.unwrap_or_else(|e| {
    tracing::warn!(%e, "nfs daemon connect failed; falling back to copy");
    create_by_copy(req)
});

let meta = std::fs::metadata(SOCKET_PATH)?; // socket must exist before connecting
if !meta.file_type().is_socket() { anyhow::bail!("{} is not a socket", SOCKET_PATH); }
// plus a liveness probe of the daemon before the call

match client.call(req).await {
    Ok(resp) => resp,
    Err(e) if e.to_string().contains("connect timed out") => {
        // short backoff, then fall back to copy strategy
        tokio::time::sleep(Duration::from_millis(200)).await;
        create_by_copy(req).await?
    }
    Err(e) => return Err(e),
}

When it happens

Trigger: Calling the NFS client (via `call`) when the daemon's unix socket exists but is not accepting: daemon hung or overloaded, backlog full, socket path stale, or the timeout is shorter than the daemon's response time.

Common situations: NFS worktree daemon crashed or was restarted while the socket file remains; system under heavy load so the daemon's event loop is stalled; connect_missing_socket_fails_quickly-style very small timeout used in production; container where the daemon isn't running.

Understand the failure class

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• Timeout waiting for IPC socket to be created
• wait failed: {body}
• grove daemon unreachable
• unix socket path too long: {}
• WebSocket connection timed out after {} seconds

AI-assisted analysis of xai-org/grok-build@bc7f02eddd (2026-08-31). Data as JSON: /api/errors/1c7e371c5220524c. Report an issue: GitHub.