What causes "capacity" in JuliusBrussee/caveman?

Serialize or throttle optimize()/retrieve() calls, e.g. with a small promise pool (<=16)

JuliusBrussee/caveman · error · MiddlewareError

capacity

capacity

Error message

capacity

What it means

MiddlewareError('capacity') is thrown by the internal exchange() when the runtime's outbound HTTP concurrency limits are exhausted: at most 16 concurrent capability/optimize/receipt fetches, or at most 1 concurrent receipt fetch. This bounds outstanding calls so a slow or hung custom transport cannot accumulate unbounded work.

Solutions

Serialize or throttle optimize()/retrieve() calls, e.g. with a small promise pool (<=16)
Share one runtime instance but cap concurrent model dispatches
Fix or replace a custom fetch transport that ignores abort signals and leaks outstanding calls
Check for a prior stuck receipt fetch before issuing another

Example fix

// before
await Promise.all(requests.map(r => runtime.optimize(r))) // >16 concurrent
// after
import pLimit from 'p-limit';
const limit = pLimit(8);
await Promise.all(requests.map(r => limit(() => runtime.optimize(r))))

Defensive patterns

Strategy: fallback

Try / catch

try { return await runtime.optimize(req); } catch (e) { if (e instanceof MiddlewareError && e.code === 'capacity') { await delay(jitter()); return sendUnoptimized(req); } throw e; }

Prevention

Cap concurrent optimize/retrieve calls (<= 8-16) with a semaphore or p-limit
Never issue concurrent receipt fetches; await the previous one
Audit custom fetch transports so they honor abort signals and never hang

When it happens

Trigger: More than 16 simultaneous optimize()/retrieve()/capabilities calls in flight (fetchesPending >= 16), or a second concurrent receipt fetch while receiptFetchesPending >= 1; especially when a custom fetch transport ignores abort and keeps calls outstanding.

Common situations: High fan-out code issuing many model requests concurrently through one shared runtime; a custom options.fetch implementation that hangs and never releases slots; a burst of optimize() calls during a slow runtime endpoint.

Related errors

AI-assisted analysis of JuliusBrussee/caveman@3ee70a1026 (2026-09-20). Data as JSON: /api/errors/e9393fb9c323a7e8. Report an issue: GitHub.

Appendix: source

Thrown at packages/sdk/typescript/src/middleware/runtime.ts:327

    if (diagnostic) { try { this.options.onDiagnostic?.({ code, cacheContinuity: 'unavailable' }); } catch { /* sink cannot break requests */ } }
    if (this.options.strict && diagnostic) throw new MiddlewareError(code);
    return { status: 'bypassed', reason: code, replacements: [], plan: null, request: null, cacheContinuity: 'unavailable' };
  }

  private async http(path: string, body: string | undefined, timeoutMs: number, signal?: AbortSignal): Promise<unknown> {
    const deadline = new AbortController();
    const timer = setTimeout(() => deadline.abort(new MiddlewareError('deadline')),Math.max(1,Math.ceil(timeoutMs)));
    try {
      return await this.exchange(path,body,AbortSignal.any([this.lifetime.signal,deadline.signal,...(signal ? [signal] : [])]));
    } finally { clearTimeout(timer); }
  }

  private async exchange(path: string, body: string | undefined, combined: AbortSignal): Promise<unknown> {
    combined.throwIfAborted();
    const headers: Record<string,string> = { 'Content-Type': 'application/json' };
    if (this.options.token) headers['Authorization'] = `Bearer ${this.options.token}`;
    const receipt = path === 'receipts';
    if (receipt ? this.receiptFetchesPending >= 1 : this.fetchesPending >= 16) throw new MiddlewareError('capacity');
    if (receipt) this.receiptFetchesPending++; else this.fetchesPending++;
    const release = () => { if (receipt) this.receiptFetchesPending--; else this.fetchesPending--; };
    const pending = Promise.resolve().then(() => this.fetcher(this.endpoint+PREFIX+path, { method: body === undefined ? 'GET' : 'POST', headers,
      ...(body !== undefined ? { body } : {}), signal: combined, redirect: 'error' }));
    // A custom transport may ignore abort. Bound its outstanding calls and
    // dispose of any late body without letting it reopen a model dispatch.
    void pending.then(response => { release(); if (combined.aborted) void response.body?.cancel().catch(() => {}); }, release);
    const response = await untilAborted(pending,combined);
    const reader = response.body?.getReader();
    if (!reader) throw new MiddlewareError('invalid_plan');
    let size = 0;
    const chunks: Uint8Array[] = [];
    try {
      for (;;) {
        const next = await untilAborted(reader.read(),combined);
        if (next.done) break;
        size += next.value.length;
        if (size > 4 << 20) throw new MiddlewareError('payload_limit');

View on GitHub (pinned to 3ee70a1026)