{"record":{"id":"816353cbe3029ee4","repo":"chroma-core/chroma","slug":"getwork-rate-limit-exhausted","errorCode":null,"errorMessage":"GetWork rate limit exhausted","messagePattern":"GetWork rate limit exhausted","errorType":"error_code","errorClass":"Status","httpStatus":429,"severity":"info","filePath":"rust/worker/src/work_queue/work_queue_server.rs","lineNumber":41,"sourceCode":"fn retry_after_ms(retry_after: Duration) -> u64 {\n    let retry_after_ms = retry_after.as_nanos().saturating_add(999_999) / 1_000_000;\n    retry_after_ms.max(1).min(u128::from(u64::MAX)) as u64\n}\n\nfn retry_at_unix_ms(retry_after: Duration, now: SystemTime) -> u64 {\n    let since_epoch = now.duration_since(UNIX_EPOCH).unwrap_or_default();\n    retry_after_ms(since_epoch.saturating_add(retry_after))\n}\n\nfn get_work_item_limit(limit: u32, max_items: u32) -> usize {\n    if max_items == 0 {\n        limit as usize\n    } else {\n        max_items as usize\n    }\n}\n\nfn resource_exhausted_status(retry_after_ms: u64) -> Status {\n    let mut status = Status::resource_exhausted(\"GetWork rate limit exhausted\");\n    status.metadata_mut().insert(\n        GET_WORK_RETRY_PUSHBACK_MS_METADATA,\n        retry_after_ms\n            .to_string()\n            .parse()\n            .expect(\"retry delay is valid ASCII metadata\"),\n    );\n    status\n}\n\nfn get_work_response(result: GetWorkResult) -> Result<Response<GetWorkResponse>, Status> {\n    get_work_response_at(result, SystemTime::now())\n}\n\nfn get_work_response_at(\n    result: GetWorkResult,\n    now: SystemTime,","sourceCodeStart":23,"sourceCodeEnd":59,"githubUrl":"https://github.com/chroma-core/chroma/blob/0dad2c71e5a6104d889048dc705f41686ae98122/rust/worker/src/work_queue/work_queue_server.rs#L23-L59","documentation":"This is a tonic gRPC Status with code RESOURCE_EXHAUSTED returned by Chroma's WorkQueueService GetWork endpoint. When the work queue manager has no work items available but signals a retry_after backoff duration, the server converts the empty result into a RESOURCE_EXHAUSTED error and attaches the retry delay in the GET_WORK_RETRY_PUSHBACK_MS_METADATA metadata key. It is a flow-control pushback telling the worker to stop polling until the indicated time rather than an operational failure.","triggerScenarios":"A worker calls the GetWork gRPC RPC and the underlying WorkQueueManager returns a GetWorkResult with an empty items list plus a Some(retry_after) Duration (get_work_response_at at work_queue_server.rs:57-66). The server converts this to resource_exhausted_status with the rounded-up retry delay in milliseconds. Any GetWork call made while the queue has no ready items and backoff is active produces this error.","commonSituations":"Idle systems where all queued work has been consumed and the queue is rate-limiting pollers; multiple workers polling the same queue competing for scarce items; aggressive polling loops that ignore the retry-pushback metadata and hammer the queue; deployments where attached functions produce work slowly relative to polling frequency.","solutions":["Read the GET_WORK_RETRY_PUSHBACK_MS_METADATA metadata from the gRPC Status and wait that many milliseconds before the next GetWork call","Implement exponential backoff with jitter in the worker polling loop instead of fixed-interval polling","Check the response body for retry_at_unix_ms when a successful response carries it, and schedule the next poll at that timestamp","Reduce the number of concurrent workers polling GetWork if the queue is persistently empty","Verify the attached functions are actually scheduled/producing work if the queue remains empty indefinitely"],"exampleFix":"// before: naive polling loop\nloop {\n    let resp = client.get_work(req).await?;\n    process(resp.items).await;\n}\n// after: honor the pushback metadata on RESOURCE_EXHAUSTED\nloop {\n    match client.get_work(req.clone()).await {\n        Ok(resp) => process(resp.into_inner().items).await,\n        Err(status) if status.code() == tonic::Code::ResourceExhausted => {\n            let delay_ms = status.metadata()\n                .get(GET_WORK_RETRY_PUSHBACK_MS_METADATA)\n                .and_then(|v| v.to_str().ok())\n                .and_then(|v| v.parse::<u64>().ok())\n                .unwrap_or(1000);\n            tokio::time::sleep(Duration::from_millis(delay_ms)).await;\n        }\n        Err(e) => return Err(e),\n    }\n}","handlingStrategy":"retry","validationCode":"// Cannot be validated before the call; queue emptiness is server state.\n// Instead, gate polling frequency client-side:\nfn should_poll(last_poll: std::time::Instant, min_interval: Duration) -> bool {\n    last_poll.elapsed() >= min_interval\n}","typeGuard":null,"tryCatchPattern":"match client.get_work(req).await {\n    Ok(resp) => { /* process items */ }\n    Err(status) if status.code() == tonic::Code::ResourceExhausted => {\n        let retry_ms = status.metadata()\n            .get(\"x-chroma-get-work-retry-pushback-ms\")\n            .and_then(|v| v.to_str().ok())\n            .and_then(|v| v.parse::<u64>().ok())\n            .unwrap_or(1000);\n        tokio::time::sleep(Duration::from_millis(retry_ms)).await;\n    }\n    Err(e) => return Err(e.into()),\n}","preventionTips":["Always read and honor the GET_WORK_RETRY_PUSHBACK_MS_METADATA metadata key on RESOURCE_EXHAUSTED errors","Treat this specific RESOURCE_EXHAUSTED status as normal flow control, not a failure — do not log it as an error or trigger alerting","Use exponential backoff with jitter for GetWork polling rather than tight fixed loops","Cap the number of concurrent workers polling the same work queue","Check successful GetWorkResponse.retry_at_unix_ms and schedule the next poll accordingly"],"tags":["grpc","rate-limit","tonic","backoff","resource-exhausted"],"backgroundTag":"rate-limit-exceeded","analyzedSha":"0dad2c71e5a6104d889048dc705f41686ae98122","analyzedAt":"2026-09-19T17:52:27.284Z","contentChangedAt":"2026-09-19T17:52:27.284Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}