t8y2/dbx · warning

agent request capacity is temporarily exhausted

Error message

agent request capacity is temporarily exhausted

What it means

The runtime bounds non-control work with a Semaphore of MAX_CONCURRENT_REQUESTS permits (runtime.rs:153). When every permit is taken — i.e. MAX_CONCURRENT_REQUESTS requests are already in flight — any new non-exempt request is immediately rejected with this error instead of being queued. Control methods (handshake, cancel_session, close_session, disconnect) are exempt via is_capacity_exempt so clients can always free capacity. The error is surfaced as a structured RpcResponse with category 'resource', retryable=true, session_disposition 'keep', so the session survives.

Source

Thrown at agents/drivers/tdengine/src/runtime.rs:112

                Ok(request) => handle_request(runtime.clone(), request).await,
                Err(error) => error_response(Value::Null, "request", None, error.into()),
            };
            responses.send(response).map_err(|_| anyhow!("TDengine response writer stopped"))?;
            while requests.join_next().await.is_some() {}
            break;
        }
        let request_permit = if parsed.as_ref().is_ok_and(|request| is_capacity_exempt(&request.method)) {
            None
        } else {
            match runtime.request_slots.clone().try_acquire_owned() {
                Ok(permit) => Some(permit),
                Err(_) => {
                    let response = match parsed {
                        Ok(request) => error_response(
                            if request.id.is_null() { json!(1) } else { request.id },
                            &request.method,
                            session_id(&request.params),
                            anyhow!("agent request capacity is temporarily exhausted"),
                        ),
                        Err(error) => error_response(Value::Null, "request", None, error.into()),
                    };
                    responses.send(response).map_err(|_| anyhow!("TDengine response writer stopped"))?;
                    continue;
                }
            }
        };
        let runtime = runtime.clone();
        let responses = responses.clone();
        requests.spawn(async move {
            let _request_permit = request_permit;
            let response = match parsed {
                Ok(request) => handle_request(runtime, request).await,
                Err(error) => error_response(Value::Null, "request", None, error.into()),
            };
            let _ = responses.send(response);
        });

View on GitHub (pinned to c0390bff16)

Solutions

1. Reduce client-side concurrency below MAX_CONCURRENT_REQUESTS (serialize or limit parallel in-flight requests)
2. Retry the request after in-flight ones complete — the error is explicitly marked retryable=true with session kept
3. Check for hung long-running queries and reclaim permits via cancel_session (which is capacity-exempt)
4. Await each response before issuing the next request if you rely on strict sequential protocol

Example fix

// before: unbounded fan-out
for table in tables {
  send({"method":"list_objects","params":{...}}); // may exceed request capacity
}
// after: bounded pipeline (e.g. 4 at a time)
for chunk in tables.chunks(4) {
  let responses = join_all(chunk.map(|t| send(list_objects(t)))).await;
  for r in responses { handle(r); } // releases permits before next chunk
}

let inFlight = 0;
const MAX_CONCURRENT = 8; // keep at or below the agent's MAX_CONCURRENT_REQUESTS
function canSend() { return inFlight < MAX_CONCURRENT; }
if (!canSend()) await waitForSlot();

function isCapacityError(res) {
  return res?.error?.data?.category === "resource"
    && /request capacity/i.test(res.error?.message ?? "");
}

async function sendWithRetry(request, retries = 5) {
  for (let i = 0; i < retries; i++) {
    const res = await sendRpc(request);
    if (isCapacityError(res)) {
      await sleep(backoff(i)); // exponential backoff; session is kept
      continue;
    }
    return res;
  }
  throw new Error("agent still at request capacity after retries");
}

When it happens

Trigger: Sending more concurrent requests than MAX_CONCURRENT_REQUESTS without waiting for responses — e.g. issuing many execute_query/list_tables calls in parallel, or long-running queries (get_explain_info, execute_transaction) holding permits while more requests stream in on stdin.

Common situations: A client pipeline fanning out metadata discovery across many tables at once; parallel dashboard queries against a busy agent; a stuck/slow query that never finishes keeps holding permits so subsequent requests get rejected; batching scripts with no concurrency cap.

Related errors

• agent operation capacity is temporarily exhausted
• agent session limit reached: {MAX_AGENT_SESSIONS}
• agent operation capacity is temporarily exhausted
• %w: %d
• JDBC Session was quarantined while waiting for a connection

AI-assisted analysis of t8y2/dbx@c0390bff16 (2026-09-05). Data as JSON: /api/errors/ed40d11cde23e410. Report an issue: GitHub.