xai-org/grok-build · error

Timed out acquiring leader lock at {}

Error message

Timed out acquiring leader lock at {}

What it means

run_leader called lock.acquire_reopen_timeout(LEADER_ACQUIRE_TIMEOUT) and received LockError::Timeout: another process held the leader lock for the entire wait period without releasing it. The process gives up and returns an error so the client can adopt whoever won the lock. This indicates a leader existed but did not hand over or die within the timeout window.

Source

Thrown at crates/codegen/xai-grok-shell/src/agent/app.rs:807

                    socket_path.display()
                );
                return Err(anyhow::anyhow!(
                    "Another leader already holds the lock at {}",
                    socket_path.display()
                ));
            }
            match lock.acquire_reopen_timeout(LEADER_ACQUIRE_TIMEOUT).await {
                Ok(()) => {
                    lock.write_pid()?;
                    debug!("Acquired leader lock after bounded wait, proceeding as leader");
                }
                Err(LockError::Timeout(_)) => {
                    info!(
                        "Timed out waiting for the leader lock ({}). Exiting so the \
                         client adopts whoever won it.",
                        socket_path.display()
                    );
                    return Err(anyhow::anyhow!(
                        "Timed out acquiring leader lock at {}",
                        socket_path.display()
                    ));
                }
                Err(e) => {
                    return Err(anyhow::anyhow!("Failed to acquire leader lock: {}", e));
                }
            }
        }
        Err(e) => return Err(anyhow::anyhow!("Failed to acquire leader lock: {}", e)),
    }
    lock.cleanup_socket()?;
    info!("Leader server starting");
    let (ipc_to_agent_tx, mut ipc_to_agent_rx) = mpsc::unbounded_channel::<String>();
    let (agent_to_ipc_tx, agent_to_ipc_rx) = mpsc::unbounded_channel::<String>();
    let (ws_to_agent_tx, mut ws_to_agent_rx) = mpsc::unbounded_channel::<String>();
    let (acp_incoming_rx, acp_incoming_tx) = simplex(MAX_BUFFER_SIZE);
    let (acp_outgoing_rx, acp_outgoing_tx) = simplex(MAX_BUFFER_SIZE);

View on GitHub (pinned to bc7f02eddd)

Solutions

1. Retry run_leader after a short delay — the lock holder usually releases and the new attempt succeeds.
2. Check whether the lock-holding process is alive and responsive; if it is hung, terminate it so the lock is released.
3. Increase LEADER_ACQUIRE_TIMEOUT if legitimate leaders routinely take longer than the current window to start.
4. Inspect the lock/socket path for a stale lockfile whose owner died; remove it once confirmed safe, then retry.
5. Avoid launching multiple agent instances from autostart scripts racing each other at login.

Example fix

// before: single attempt, hard failure
run_agent_command(...).await?;

// after: tolerate transient contention with a bounded retry
for attempt in 0..3 {
    match run_agent_command(...).await {
        Ok(v) => return Ok(v),
        Err(e) if e.to_string().contains("Timed out acquiring leader lock") => {
            tokio::time::sleep(Duration::from_millis(500)).await;
        }
        Err(e) => return Err(e),
    }
}

let socket_path = agent_socket_path();
if crate::leader::listener_is_ready(&socket_path) {
    return Ok(()); // leader exists; no need to wait on the lock at all
}

fn is_lock_timeout_err(e: &anyhow::Error) -> bool {
    e.to_string().contains("Timed out acquiring leader lock")
}

for _ in 0..3 {
    match run_agent_command(...).await {
        Err(e) if is_lock_timeout_err(&e) => tokio::time::sleep(Duration::from_secs(1)).await,
        other => return other,
    }
}
Err(anyhow::anyhow!("leader lock still contended after retries"))

When it happens

Trigger: Calling run_leader while another process holds the flock/lockfile and `listener_is_ready` was false at check time (leader was mid-startup, wedged, or its lock outlived its socket); the wait exceeds LEADER_ACQUIRE_TIMEOUT.

Common situations: Two agents started nearly simultaneously and the loser waited past the timeout; the current leader is hung (event loop stalled) so it never releases the lock; very long leader startup exceeding LEADER_ACQUIRE_TIMEOUT; NFS/network filesystem where flock release is delayed.

Understand the failure class

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• Failed to acquire leader lock: {}
• Another leader already holds the lock at {}
• Timeout waiting for IPC socket to be created
• connect timed out
• wait failed: {body}

AI-assisted analysis of xai-org/grok-build@bc7f02eddd (2026-08-31). Data as JSON: /api/errors/67526f94eda740ab. Report an issue: GitHub.