paperclipai/paperclip · warning · PhotonError

quota

quota

Error message

Photon recovery is temporarily rate limited

What it means

When the Photon catchup stream ends with gRPC status RESOURCE_EXHAUSTED, the transport converts it into a PhotonError with code 'quota'. Photon is rate limiting recovery/catchup traffic (per-line or per-tenant throttling), so the read cannot proceed right now. Unlike 'credentials', this is transient: waiting and retrying with backoff is the correct response.

Solutions

1. Retry catchUp() after an exponential-backoff delay with jitter instead of immediately reconnecting
2. Reduce catchup frequency: reuse a single long-lived transport and only reconnect on real failures
3. Persist a smaller cursor gap (catch up more often) to lighten each replay
4. Raise the Photon rate limit/quota for this tenant if the workload legitimately requires it

Example fix

// before: tight reconnect loop
while (true) stream = transport.catchUp(seq);
// after: backoff on quota errors
let delay = 1000;
while (true) {
  try { stream = transport.catchUp(seq); break; }
  catch (e) {
    if (e instanceof PhotonError && e.code === "quota") {
      await sleep(delay + Math.random() * delay);
      delay = Math.min(delay * 2, 60_000);
      continue;
    }
    throw e;
  }
}

// gauge catchup pressure before opening the stream
const gap = lastKnownSequence != null ? currentSequence - lastKnownSequence : Infinity;
if (gap > 100_000) console.warn("Large recovery gap; consider staged catchup to avoid quota throttling");

function isPhotonQuotaError(e: unknown): e is PhotonError {
  return e instanceof PhotonError && e.code === "quota";
}

let delay = 1_000;
for (;;) {
  try { return transport.catchUp(seq); }
  catch (e) {
    if (isPhotonQuotaError(e)) {
      await sleep(delay + Math.random() * delay); // exponential backoff + jitter
      delay = Math.min(delay * 2, 60_000);
      continue;
    }
    throw e;
  }
}

When it happens

Trigger: Calling catchUp() (via receive()) while the Photon server is throttling the caller: too many concurrent catchup streams, repeated rapid reconnects after failures, large history replays hammering the stream, or a low tenant quota.

Common situations: Crash-looping instances that reconnect in a tight loop and trip the server rate limiter; a burst of catchup requests after a shared server outage ends; many agents of the same company replaying large history gaps simultaneously.

Related errors

• credentials
• dropping batch after attempt(s); event(s) lost
• railway_rate_limited
• ACPX recovery identity does not match the persisted runtime…
• Anthropic Managed Agents request failed with HTTP

AI-assisted analysis of paperclipai/paperclip@3f1d897a7c (2026-09-18). Data as JSON: /api/errors/891e2122dfaea11f. Report an issue: GitHub.

Appendix: source

Thrown at server/src/services/photon/recovery-transport.ts:163

        }
      } catch (error) {
        if (controller.signal.aborted) return;
        const code = (error as { code?: number }).code;
        if (code === status.OUT_OF_RANGE || code === status.FAILED_PRECONDITION)
          throw new PhotonError(
            "history_gap",
            "Photon cannot recover the saved cursor; reconnect after reviewing the history gap",
          );
        if (
          code === status.UNAUTHENTICATED ||
          code === status.PERMISSION_DENIED
        )
          throw new PhotonError(
            "credentials",
            "Photon rejected the selected line credentials; reconnect the channel",
          );
        if (code === status.RESOURCE_EXHAUSTED)
          throw new PhotonError(
            "quota",
            "Photon recovery is temporarily rate limited",
          );
        throw photonFailure(error);
      } finally {
        controller.signal.removeEventListener("abort", abort);
        call.cancel();
      }
    }
    return new TypedEventStream(receive(), async () => controller.abort());
  }
  close(): void {
    this.client.close();
  }
}

View on GitHub (pinned to 3f1d897a7c)