{"record":{"id":"67526f94eda740ab","repo":"xai-org/grok-build","slug":"timed-out-acquiring-leader-lock-at","errorCode":null,"errorMessage":"Timed out acquiring leader lock at {}","messagePattern":"Timed out acquiring leader lock at (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"crates/codegen/xai-grok-shell/src/agent/app.rs","lineNumber":807,"sourceCode":"                    socket_path.display()\n                );\n                return Err(anyhow::anyhow!(\n                    \"Another leader already holds the lock at {}\",\n                    socket_path.display()\n                ));\n            }\n            match lock.acquire_reopen_timeout(LEADER_ACQUIRE_TIMEOUT).await {\n                Ok(()) => {\n                    lock.write_pid()?;\n                    debug!(\"Acquired leader lock after bounded wait, proceeding as leader\");\n                }\n                Err(LockError::Timeout(_)) => {\n                    info!(\n                        \"Timed out waiting for the leader lock ({}). Exiting so the \\\n                         client adopts whoever won it.\",\n                        socket_path.display()\n                    );\n                    return Err(anyhow::anyhow!(\n                        \"Timed out acquiring leader lock at {}\",\n                        socket_path.display()\n                    ));\n                }\n                Err(e) => {\n                    return Err(anyhow::anyhow!(\"Failed to acquire leader lock: {}\", e));\n                }\n            }\n        }\n        Err(e) => return Err(anyhow::anyhow!(\"Failed to acquire leader lock: {}\", e)),\n    }\n    lock.cleanup_socket()?;\n    info!(\"Leader server starting\");\n    let (ipc_to_agent_tx, mut ipc_to_agent_rx) = mpsc::unbounded_channel::<String>();\n    let (agent_to_ipc_tx, agent_to_ipc_rx) = mpsc::unbounded_channel::<String>();\n    let (ws_to_agent_tx, mut ws_to_agent_rx) = mpsc::unbounded_channel::<String>();\n    let (acp_incoming_rx, acp_incoming_tx) = simplex(MAX_BUFFER_SIZE);\n    let (acp_outgoing_rx, acp_outgoing_tx) = simplex(MAX_BUFFER_SIZE);","sourceCodeStart":789,"sourceCodeEnd":825,"githubUrl":"https://github.com/xai-org/grok-build/blob/bc7f02eddd3d84085849dc19ed216f11c23b0571/crates/codegen/xai-grok-shell/src/agent/app.rs#L789-L825","documentation":"run_leader called lock.acquire_reopen_timeout(LEADER_ACQUIRE_TIMEOUT) and received LockError::Timeout: another process held the leader lock for the entire wait period without releasing it. The process gives up and returns an error so the client can adopt whoever won the lock. This indicates a leader existed but did not hand over or die within the timeout window.","triggerScenarios":"Calling run_leader while another process holds the flock/lockfile and `listener_is_ready` was false at check time (leader was mid-startup, wedged, or its lock outlived its socket); the wait exceeds LEADER_ACQUIRE_TIMEOUT.","commonSituations":"Two agents started nearly simultaneously and the loser waited past the timeout; the current leader is hung (event loop stalled) so it never releases the lock; very long leader startup exceeding LEADER_ACQUIRE_TIMEOUT; NFS/network filesystem where flock release is delayed.","solutions":["Retry run_leader after a short delay — the lock holder usually releases and the new attempt succeeds.","Check whether the lock-holding process is alive and responsive; if it is hung, terminate it so the lock is released.","Increase LEADER_ACQUIRE_TIMEOUT if legitimate leaders routinely take longer than the current window to start.","Inspect the lock/socket path for a stale lockfile whose owner died; remove it once confirmed safe, then retry.","Avoid launching multiple agent instances from autostart scripts racing each other at login."],"exampleFix":"// before: single attempt, hard failure\nrun_agent_command(...).await?;\n\n// after: tolerate transient contention with a bounded retry\nfor attempt in 0..3 {\n    match run_agent_command(...).await {\n        Ok(v) => return Ok(v),\n        Err(e) if e.to_string().contains(\"Timed out acquiring leader lock\") => {\n            tokio::time::sleep(Duration::from_millis(500)).await;\n        }\n        Err(e) => return Err(e),\n    }\n}","handlingStrategy":"retry","validationCode":"let socket_path = agent_socket_path();\nif crate::leader::listener_is_ready(&socket_path) {\n    return Ok(()); // leader exists; no need to wait on the lock at all\n}","typeGuard":"fn is_lock_timeout_err(e: &anyhow::Error) -> bool {\n    e.to_string().contains(\"Timed out acquiring leader lock\")\n}","tryCatchPattern":"for _ in 0..3 {\n    match run_agent_command(...).await {\n        Err(e) if is_lock_timeout_err(&e) => tokio::time::sleep(Duration::from_secs(1)).await,\n        other => return other,\n    }\n}\nErr(anyhow::anyhow!(\"leader lock still contended after retries\"))","preventionTips":["Probe listener_is_ready before waiting on the lock to skip the timeout path entirely.","Size LEADER_ACQUIRE_TIMEOUT above your worst-case leader startup time.","Avoid racing multiple autostart instances at login; stagger or deduplicate launches.","Monitor for hung leaders — repeated timeouts indicate a wedged holder that must be killed."],"tags":["ipc","leader-election","file-lock","timeout"],"backgroundTag":"leader-lock-timeout","analyzedSha":"bc7f02eddd3d84085849dc19ed216f11c23b0571","analyzedAt":"2026-08-31T04:59:42.031Z","schemaVersion":2},"datasetVersion":"2026-08-31T09:17:48.483Z"}