{"record":{"id":"83217cd35567b45e","repo":"xai-org/grok-build","slug":"failed-to-acquire-leader-lock","errorCode":null,"errorMessage":"Failed to acquire leader lock: {}","messagePattern":"Failed to acquire leader lock: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"crates/codegen/xai-grok-shell/src/agent/app.rs","lineNumber":813,"sourceCode":"            }\n            match lock.acquire_reopen_timeout(LEADER_ACQUIRE_TIMEOUT).await {\n                Ok(()) => {\n                    lock.write_pid()?;\n                    debug!(\"Acquired leader lock after bounded wait, proceeding as leader\");\n                }\n                Err(LockError::Timeout(_)) => {\n                    info!(\n                        \"Timed out waiting for the leader lock ({}). Exiting so the \\\n                         client adopts whoever won it.\",\n                        socket_path.display()\n                    );\n                    return Err(anyhow::anyhow!(\n                        \"Timed out acquiring leader lock at {}\",\n                        socket_path.display()\n                    ));\n                }\n                Err(e) => {\n                    return Err(anyhow::anyhow!(\"Failed to acquire leader lock: {}\", e));\n                }\n            }\n        }\n        Err(e) => return Err(anyhow::anyhow!(\"Failed to acquire leader lock: {}\", e)),\n    }\n    lock.cleanup_socket()?;\n    info!(\"Leader server starting\");\n    let (ipc_to_agent_tx, mut ipc_to_agent_rx) = mpsc::unbounded_channel::<String>();\n    let (agent_to_ipc_tx, agent_to_ipc_rx) = mpsc::unbounded_channel::<String>();\n    let (ws_to_agent_tx, mut ws_to_agent_rx) = mpsc::unbounded_channel::<String>();\n    let (acp_incoming_rx, acp_incoming_tx) = simplex(MAX_BUFFER_SIZE);\n    let (acp_outgoing_rx, acp_outgoing_tx) = simplex(MAX_BUFFER_SIZE);\n    let incoming = acp_incoming_rx.compat();\n    let outgoing = acp_outgoing_tx.compat_write();\n    let acp_incoming_tx = Arc::new(TokioMutex::new(acp_incoming_tx));\n    let cancel = CancellationToken::new();\n    let (ready_tx, ready_rx) = watch::channel(false);\n    let (shutdown_tx, _shutdown_reason_rx) = watch::channel(ShutdownReason::Manual);","sourceCodeStart":795,"sourceCodeEnd":831,"githubUrl":"https://github.com/xai-org/grok-build/blob/bc7f02eddd3d84085849dc19ed216f11c23b0571/crates/codegen/xai-grok-shell/src/agent/app.rs#L795-L831","documentation":"While waiting for the leader lock, acquire_reopen_timeout returned a LockError that was not Timeout (e.g. I/O failure on the lockfile or an unexpected lock error), and run_leader wrapped it with anyhow as \"Failed to acquire leader lock: {}\". Unlike the timeout case, this means the lock machinery itself failed rather than merely being contended.","triggerScenarios":"lock.acquire_reopen_timeout(LEADER_ACQUIRE_TIMEOUT) returns Err(LockError) that does not match LockError::Timeout — typically an OS-level flock/open error on the lock file inside the inner match arm of run_leader.","commonSituations":"Read-only or full filesystem containing the lock path; permission denied on the runtime/lock directory; XDG_RUNTIME_DIR unset or pointing to an unwritable location; lockfile deleted or replaced underneath the holder; SELinux/AppArmor denying file creation.","solutions":["Read the wrapped source error after the colon to identify the underlying OS error and fix that cause.","Ensure the lock directory exists and is writable: check XDG_RUNTIME_DIR and `mkdir -p` / chown the path if needed.","Free disk space or remount the filesystem read-write if the error indicates ENOSPC/EROFS.","Remove a corrupt or stale lockfile after confirming no live process owns it, then retry.","Run with elevated debug logging to capture the exact io::Error kind from the lock crate."],"exampleFix":"// before: unwritable runtime dir crashes acquire\n// XDG_RUNTIME_DIR=/nonexistent agent run\n\n// after: validate/prepare the runtime dir before launching\nlet runtime = std::env::var(\"XDG_RUNTIME_DIR\").unwrap_or(format!(\"/tmp/run-{}\", uid));\nstd::fs::create_dir_all(&runtime)?;","handlingStrategy":"try-catch","validationCode":"let lock_dir = lock_path.parent().expect(\"lock path has parent\");\nif !lock_dir.exists() {\n    std::fs::create_dir_all(lock_dir)?;\n}\nlet probe = lock_dir.join(\".write-probe\");\nstd::fs::write(&probe, b\"ok\").map_err(|e| anyhow!(\"runtime dir {} not writable: {}\", lock_dir.display(), e))?;\nlet _ = std::fs::remove_file(&probe);","typeGuard":"fn is_non_timeout_lock_err(e: &anyhow::Error) -> bool {\n    let s = e.to_string();\n    s.starts_with(\"Failed to acquire leader lock\")\n        && !s.contains(\"Timed out acquiring leader lock\")\n}","tryCatchPattern":"if let Err(e) = run_agent_command(...).await {\n    if is_non_timeout_lock_err(&e) {\n        error!(\"leader lock unusable, inspecting cause: {:#}\", e);\n        // surface e.chain() to the user instead of a bare message\n    }\n    return Err(e);\n}","preventionTips":["Validate that XDG_RUNTIME_DIR (or the lock dir) exists and is writable before starting the agent.","Check disk space and mount flags (no ro, no noexec-on-runtime) in deployment scripts.","Log the full anyhow error chain ({:#}) so the underlying io::Error kind is never lost.","Keep the lock file on a local filesystem, not NFS, to avoid flock anomalies."],"tags":["ipc","leader-election","file-lock","filesystem"],"backgroundTag":"leader-lock-acquire-failed","analyzedSha":"bc7f02eddd3d84085849dc19ed216f11c23b0571","analyzedAt":"2026-08-31T04:59:42.031Z","schemaVersion":2},"datasetVersion":"2026-08-31T09:17:48.483Z"}