What causes "timed out waiting {}s for git object database permit" in xai-org/grok-build?

Increase the acquire_wait timeout configuration to tolerate queueing.

timed out waiting {}s for git object database permit

Error message

timed out waiting {}s for git object database permit

What it means

OdbHandle::acquire bounded-waits for a permit on the git object database's concurrency-limiting semaphore using the configured acquire_wait duration. When the future times out (tokio::time::timeout elapsed), this error is returned, including the wait duration in seconds. It indicates the object database is saturated with concurrent users.

Source

Thrown at crates/codegen/xai-grok-workspace/src/git_odb.rs:67

    pub fn new(permits: usize, acquire_wait: Duration) -> Self {
        Self {
            inner: Arc::new(OdbLimiterInner {
                sem: Arc::new(Semaphore::new(permits.max(1))),
                acquire_wait,
            }),
        }
    }

    pub async fn acquire(&self) -> Result<OdbPermit> {
        match tokio::time::timeout(
            self.inner.acquire_wait,
            self.inner.sem.clone().acquire_owned(),
        )
        .await
        {
            Ok(Ok(permit)) => Ok(OdbPermit { _permit: permit }),
            Ok(Err(_)) => Err(anyhow!("git object database semaphore closed")),
            Err(_) => Err(anyhow!(
                "timed out waiting {}s for git object database permit",
                self.inner.acquire_wait.as_secs()
            )),
        }
    }

    pub fn try_acquire(&self) -> Option<OdbPermit> {
        self.inner
            .sem
            .clone()
            .try_acquire_owned()
            .ok()
            .map(|permit| OdbPermit { _permit: permit })
    }
}

#[cfg(test)]
mod tests {

View on GitHub (pinned to bc7f02eddd)

Solutions

Increase the acquire_wait timeout configuration to tolerate queueing.
Reduce concurrency of callers doing git object reads (batch/serialize heavy scans).
Investigate permit leaks — operations that hold permits longer than expected or never release them on early-return paths.
Retry with backoff; the saturation is often transient.

Example fix

// before: default (too short) wait under heavy load
let permit = odb.acquire().await?;
// after: raise the wait budget
let odb = odb.with_acquire_wait(Duration::from_secs(60));
let permit = odb.acquire().await?;

Defensive patterns

Strategy: retry

Validate before calling

let in_flight = odb.permits_in_use();
if in_flight >= odb.capacity() {
    anyhow::bail!("git odb saturated ({in_flight}/{}) — reduce concurrency first", odb.capacity());
}

Try / catch

let permit = loop {
    match odb.acquire().await {
        Ok(p) => break Ok(p),
        Err(e) if e.to_string().contains("timed out") => {
            backoff.wait();
            if backoff.exhausted() { break Err(e); }
        }
        Err(e) => break Err(e),
    }
}?;

Prevention

Configure acquire_wait generously for large-repo batch workloads
Limit parallel git object readers to the semaphore capacity
Hold permits for the shortest possible scope (drop immediately after the read)
Watch for leaked permits via metrics/logging in early-return paths

When it happens

Trigger: Calling acquire when the number of concurrent permit holders has reached the semaphore capacity for longer than acquire_wait — e.g. many parallel git object reads (large history scans, batch operations) all holding permits.

Common situations: Large repos with many simultaneous file-history or blame queries; long-running operations holding permits while others queue; acquire_wait configured too small for the workload; a leaked permit path keeping capacity occupied.

Understand the failure class

Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

AI-assisted analysis of xai-org/grok-build@bc7f02eddd (2026-08-31). Data as JSON: /api/errors/9cfbd0e923705392. Report an issue: GitHub.