What causes "Failed to acquire leader lock: {}" in xai-org/grok-build?

Read the wrapped source error after the colon to identify the underlying OS error and fix that cause.

Failed to acquire leader lock: {}

Error message

Failed to acquire leader lock: {}

What it means

While waiting for the leader lock, acquire_reopen_timeout returned a LockError that was not Timeout (e.g. I/O failure on the lockfile or an unexpected lock error), and run_leader wrapped it with anyhow as "Failed to acquire leader lock: {}". Unlike the timeout case, this means the lock machinery itself failed rather than merely being contended.

Source

Thrown at crates/codegen/xai-grok-shell/src/agent/app.rs:813

            }
            match lock.acquire_reopen_timeout(LEADER_ACQUIRE_TIMEOUT).await {
                Ok(()) => {
                    lock.write_pid()?;
                    debug!("Acquired leader lock after bounded wait, proceeding as leader");
                }
                Err(LockError::Timeout(_)) => {
                    info!(
                        "Timed out waiting for the leader lock ({}). Exiting so the \
                         client adopts whoever won it.",
                        socket_path.display()
                    );
                    return Err(anyhow::anyhow!(
                        "Timed out acquiring leader lock at {}",
                        socket_path.display()
                    ));
                }
                Err(e) => {
                    return Err(anyhow::anyhow!("Failed to acquire leader lock: {}", e));
                }
            }
        }
        Err(e) => return Err(anyhow::anyhow!("Failed to acquire leader lock: {}", e)),
    }
    lock.cleanup_socket()?;
    info!("Leader server starting");
    let (ipc_to_agent_tx, mut ipc_to_agent_rx) = mpsc::unbounded_channel::<String>();
    let (agent_to_ipc_tx, agent_to_ipc_rx) = mpsc::unbounded_channel::<String>();
    let (ws_to_agent_tx, mut ws_to_agent_rx) = mpsc::unbounded_channel::<String>();
    let (acp_incoming_rx, acp_incoming_tx) = simplex(MAX_BUFFER_SIZE);
    let (acp_outgoing_rx, acp_outgoing_tx) = simplex(MAX_BUFFER_SIZE);
    let incoming = acp_incoming_rx.compat();
    let outgoing = acp_outgoing_tx.compat_write();
    let acp_incoming_tx = Arc::new(TokioMutex::new(acp_incoming_tx));
    let cancel = CancellationToken::new();
    let (ready_tx, ready_rx) = watch::channel(false);
    let (shutdown_tx, _shutdown_reason_rx) = watch::channel(ShutdownReason::Manual);

View on GitHub (pinned to bc7f02eddd)

Solutions

Read the wrapped source error after the colon to identify the underlying OS error and fix that cause.
Ensure the lock directory exists and is writable: check XDG_RUNTIME_DIR and `mkdir -p` / chown the path if needed.
Free disk space or remount the filesystem read-write if the error indicates ENOSPC/EROFS.
Remove a corrupt or stale lockfile after confirming no live process owns it, then retry.
Run with elevated debug logging to capture the exact io::Error kind from the lock crate.

Example fix

// before: unwritable runtime dir crashes acquire
// XDG_RUNTIME_DIR=/nonexistent agent run

// after: validate/prepare the runtime dir before launching
let runtime = std::env::var("XDG_RUNTIME_DIR").unwrap_or(format!("/tmp/run-{}", uid));
std::fs::create_dir_all(&runtime)?;

Defensive patterns

Strategy: try-catch

Validate before calling

let lock_dir = lock_path.parent().expect("lock path has parent");
if !lock_dir.exists() {
    std::fs::create_dir_all(lock_dir)?;
}
let probe = lock_dir.join(".write-probe");
std::fs::write(&probe, b"ok").map_err(|e| anyhow!("runtime dir {} not writable: {}", lock_dir.display(), e))?;
let _ = std::fs::remove_file(&probe);

Type guard

fn is_non_timeout_lock_err(e: &anyhow::Error) -> bool {
    let s = e.to_string();
    s.starts_with("Failed to acquire leader lock")
        && !s.contains("Timed out acquiring leader lock")
}

Try / catch

if let Err(e) = run_agent_command(...).await {
    if is_non_timeout_lock_err(&e) {
        error!("leader lock unusable, inspecting cause: {:#}", e);
        // surface e.chain() to the user instead of a bare message
    }
    return Err(e);
}

Prevention

Validate that XDG_RUNTIME_DIR (or the lock dir) exists and is writable before starting the agent.
Check disk space and mount flags (no ro, no noexec-on-runtime) in deployment scripts.
Log the full anyhow error chain ({:#}) so the underlying io::Error kind is never lost.
Keep the lock file on a local filesystem, not NFS, to avoid flock anomalies.

When it happens

Trigger: lock.acquire_reopen_timeout(LEADER_ACQUIRE_TIMEOUT) returns Err(LockError) that does not match LockError::Timeout — typically an OS-level flock/open error on the lock file inside the inner match arm of run_leader.

Common situations: Read-only or full filesystem containing the lock path; permission denied on the runtime/lock directory; XDG_RUNTIME_DIR unset or pointing to an unwritable location; lockfile deleted or replaced underneath the holder; SELinux/AppArmor denying file creation.

Related errors

AI-assisted analysis of xai-org/grok-build@bc7f02eddd (2026-08-31). Data as JSON: /api/errors/83217cd35567b45e. Report an issue: GitHub.