zeroclaw-labs/zeroclaw · error · anyhow::Error

Timed out waiting for auth profile lock at {}

Error message

Timed out waiting for auth profile lock at {}

What it means

`acquire_lock` tries to create `auth-profiles.lock` with create-new semantics, polls every 50 ms, and gives up after 10 s (LOCK_TIMEOUT_MS). Every store operation — load, list, upsert, remove, set_active, clear_active — takes this lock, so one long-lived holder blocks all auth profile I/O across processes. The guard deletes the file on Drop, but a SIGKILLed process leaks it until removed.

Source

Thrown at crates/zeroclaw-providers/src/auth/profiles.rs:544

                                    .with_attrs(::serde_json::json!({"e": format!("{:?}", e)})),
                                    "Failed to remove auth profile lock file: "
                                );
                            })
                            .ok();
                        return Err(e).with_context(|| {
                            format!(
                                "Failed to write auth profile lock at {}",
                                self.lock_path.display()
                            )
                        });
                    }
                    return Ok(AuthProfileLockGuard {
                        lock_path: self.lock_path.clone(),
                    });
                }
                Err(e) if e.kind() == std::io::ErrorKind::AlreadyExists => {
                    if waited >= LOCK_TIMEOUT_MS {
                        anyhow::bail!(
                            "Timed out waiting for auth profile lock at {}",
                            self.lock_path.display()
                        );
                    }
                    sleep(Duration::from_millis(LOCK_WAIT_MS)).await;
                    waited = waited.saturating_add(LOCK_WAIT_MS);
                }
                Err(e) => {
                    return Err(e).with_context(|| {
                        format!(
                            "Failed to create auth profile lock at {}",
                            self.lock_path.display()
                        )
                    });
                }
            }
        }
    }

View on GitHub (pinned to 88bb9c8533)

Solutions

  1. Retry once after a short wait — a legitimate holder usually finishes quickly
  2. Inspect the lock file: its first line is `pid=<n>`; if that pid is not alive, delete the stale lock file
  3. Cache loaded profile data in-process instead of calling `load()` per request
  4. Keep lock-guard lifetimes short: drop the store result guard before unrelated awaits

Example fix

// before: per-request store hit under a busy daemon
let data = store.load().await?; // "Timed out waiting for auth profile lock at ..."

// after: on timeout, clear a dead-holder lock and retry once
let data = match store.load().await {
    Ok(d) => d,
    Err(e) if e.to_string().contains("Timed out waiting for auth profile lock") => {
        if lock_holder_dead(&lock_path).await { tokio::fs::remove_file(&lock_path).await?; }
        store.load().await?
    }
    Err(e) => return Err(e),
};
Defensive patterns

Strategy: retry

Validate before calling

async fn lock_holder_is_dead(path: &std::path::Path) -> bool {
    let body = tokio::fs::read_to_string(path).await.unwrap_or_default();
    let pid = body.trim().strip_prefix("pid=").and_then(|p| p.parse::<u32>().ok());
    !std::path::Path::new(&format!("/proc/{}", pid.unwrap_or(0))).exists()
}

Try / catch

match store.load().await {
    Ok(d) => d,
    Err(e) if e.to_string().contains("Timed out waiting for auth profile lock") => {
        if lock_holder_is_dead(&lock_path).await {
            tokio::fs::remove_file(&lock_path).await?; // stale lock from a killed process
        }
        store.load().await? // one retry
    }
    Err(e) => return Err(e),
}

Prevention

When it happens

Trigger: A zeroclaw daemon is mid token-refresh (holding the lock) while the CLI also touches the store; a previous run was killed with SIGKILL leaving a stale lock file; tight loops doing per-request `load()` calls extend hold time past the 10 s budget.

Common situations: CLI plus background service on the same machine; test suites that drop guards late; crash leftovers from forced shutdowns.

Understand the failure class

Related errors

AI-assisted analysis of zeroclaw-labs/zeroclaw@88bb9c8533 (2026-08-23). Data as JSON: /api/errors/50842c791d244b32. Report an issue: GitHub.