chroma-core/chroma · info · Status

GetWork rate limit exhausted

Error message

GetWork rate limit exhausted

What it means

This is a tonic gRPC Status with code RESOURCE_EXHAUSTED returned by Chroma's WorkQueueService GetWork endpoint. When the work queue manager has no work items available but signals a retry_after backoff duration, the server converts the empty result into a RESOURCE_EXHAUSTED error and attaches the retry delay in the GET_WORK_RETRY_PUSHBACK_MS_METADATA metadata key. It is a flow-control pushback telling the worker to stop polling until the indicated time rather than an operational failure.

Solutions

1. Read the GET_WORK_RETRY_PUSHBACK_MS_METADATA metadata from the gRPC Status and wait that many milliseconds before the next GetWork call
2. Implement exponential backoff with jitter in the worker polling loop instead of fixed-interval polling
3. Check the response body for retry_at_unix_ms when a successful response carries it, and schedule the next poll at that timestamp
4. Reduce the number of concurrent workers polling GetWork if the queue is persistently empty
5. Verify the attached functions are actually scheduled/producing work if the queue remains empty indefinitely

Example fix

// before: naive polling loop
loop {
    let resp = client.get_work(req).await?;
    process(resp.items).await;
}
// after: honor the pushback metadata on RESOURCE_EXHAUSTED
loop {
    match client.get_work(req.clone()).await {
        Ok(resp) => process(resp.into_inner().items).await,
        Err(status) if status.code() == tonic::Code::ResourceExhausted => {
            let delay_ms = status.metadata()
                .get(GET_WORK_RETRY_PUSHBACK_MS_METADATA)
                .and_then(|v| v.to_str().ok())
                .and_then(|v| v.parse::<u64>().ok())
                .unwrap_or(1000);
            tokio::time::sleep(Duration::from_millis(delay_ms)).await;
        }
        Err(e) => return Err(e),
    }
}

// Cannot be validated before the call; queue emptiness is server state.
// Instead, gate polling frequency client-side:
fn should_poll(last_poll: std::time::Instant, min_interval: Duration) -> bool {
    last_poll.elapsed() >= min_interval
}

match client.get_work(req).await {
    Ok(resp) => { /* process items */ }
    Err(status) if status.code() == tonic::Code::ResourceExhausted => {
        let retry_ms = status.metadata()
            .get("x-chroma-get-work-retry-pushback-ms")
            .and_then(|v| v.to_str().ok())
            .and_then(|v| v.parse::<u64>().ok())
            .unwrap_or(1000);
        tokio::time::sleep(Duration::from_millis(retry_ms)).await;
    }
    Err(e) => return Err(e.into()),
}

When it happens

Trigger: A worker calls the GetWork gRPC RPC and the underlying WorkQueueManager returns a GetWorkResult with an empty items list plus a Some(retry_after) Duration (get_work_response_at at work_queue_server.rs:57-66). The server converts this to resource_exhausted_status with the rounded-up retry delay in milliseconds. Any GetWork call made while the queue has no ready items and backoff is active produces this error.

Common situations: Idle systems where all queued work has been consumed and the queue is rate-limiting pollers; multiple workers polling the same queue competing for scarce items; aggressive polling loops that ignore the retry-pushback metadata and hammer the queue; deployments where attached functions produce work slowly relative to polling frequency.

Related errors

• Backoff and retry
• Failed to generate embeddings
• not implemented
• Rate limit exceeded

AI-assisted analysis of chroma-core/chroma@0dad2c71e5 (2026-09-19). Data as JSON: /api/errors/816353cbe3029ee4. Report an issue: GitHub.

Appendix: source

Thrown at rust/worker/src/work_queue/work_queue_server.rs:41

fn retry_after_ms(retry_after: Duration) -> u64 {
    let retry_after_ms = retry_after.as_nanos().saturating_add(999_999) / 1_000_000;
    retry_after_ms.max(1).min(u128::from(u64::MAX)) as u64
}

fn retry_at_unix_ms(retry_after: Duration, now: SystemTime) -> u64 {
    let since_epoch = now.duration_since(UNIX_EPOCH).unwrap_or_default();
    retry_after_ms(since_epoch.saturating_add(retry_after))
}

fn get_work_item_limit(limit: u32, max_items: u32) -> usize {
    if max_items == 0 {
        limit as usize
    } else {
        max_items as usize
    }
}

fn resource_exhausted_status(retry_after_ms: u64) -> Status {
    let mut status = Status::resource_exhausted("GetWork rate limit exhausted");
    status.metadata_mut().insert(
        GET_WORK_RETRY_PUSHBACK_MS_METADATA,
        retry_after_ms
            .to_string()
            .parse()
            .expect("retry delay is valid ASCII metadata"),
    );
    status
}

fn get_work_response(result: GetWorkResult) -> Result<Response<GetWorkResponse>, Status> {
    get_work_response_at(result, SystemTime::now())
}

fn get_work_response_at(
    result: GetWorkResult,
    now: SystemTime,

View on GitHub (pinned to 0dad2c71e5)