{"record":{"id":"891e2122dfaea11f","repo":"paperclipai/paperclip","slug":"quota-photon-recovery-is-temporarily-rate-limited","errorCode":"quota","errorMessage":"Photon recovery is temporarily rate limited","messagePattern":"Photon recovery is temporarily rate limited","errorType":"error_code","errorClass":"PhotonError","httpStatus":null,"severity":"warning","filePath":"server/src/services/photon/recovery-transport.ts","lineNumber":163,"sourceCode":"        }\n      } catch (error) {\n        if (controller.signal.aborted) return;\n        const code = (error as { code?: number }).code;\n        if (code === status.OUT_OF_RANGE || code === status.FAILED_PRECONDITION)\n          throw new PhotonError(\n            \"history_gap\",\n            \"Photon cannot recover the saved cursor; reconnect after reviewing the history gap\",\n          );\n        if (\n          code === status.UNAUTHENTICATED ||\n          code === status.PERMISSION_DENIED\n        )\n          throw new PhotonError(\n            \"credentials\",\n            \"Photon rejected the selected line credentials; reconnect the channel\",\n          );\n        if (code === status.RESOURCE_EXHAUSTED)\n          throw new PhotonError(\n            \"quota\",\n            \"Photon recovery is temporarily rate limited\",\n          );\n        throw photonFailure(error);\n      } finally {\n        controller.signal.removeEventListener(\"abort\", abort);\n        call.cancel();\n      }\n    }\n    return new TypedEventStream(receive(), async () => controller.abort());\n  }\n  close(): void {\n    this.client.close();\n  }\n}\n","sourceCodeStart":145,"sourceCodeEnd":179,"githubUrl":"https://github.com/paperclipai/paperclip/blob/3f1d897a7c018d76563a21c6e39c3c9b03933622/server/src/services/photon/recovery-transport.ts#L145-L179","documentation":"When the Photon catchup stream ends with gRPC status RESOURCE_EXHAUSTED, the transport converts it into a PhotonError with code 'quota'. Photon is rate limiting recovery/catchup traffic (per-line or per-tenant throttling), so the read cannot proceed right now. Unlike 'credentials', this is transient: waiting and retrying with backoff is the correct response.","triggerScenarios":"Calling catchUp() (via receive()) while the Photon server is throttling the caller: too many concurrent catchup streams, repeated rapid reconnects after failures, large history replays hammering the stream, or a low tenant quota.","commonSituations":"Crash-looping instances that reconnect in a tight loop and trip the server rate limiter; a burst of catchup requests after a shared server outage ends; many agents of the same company replaying large history gaps simultaneously.","solutions":["Retry catchUp() after an exponential-backoff delay with jitter instead of immediately reconnecting","Reduce catchup frequency: reuse a single long-lived transport and only reconnect on real failures","Persist a smaller cursor gap (catch up more often) to lighten each replay","Raise the Photon rate limit/quota for this tenant if the workload legitimately requires it"],"exampleFix":"// before: tight reconnect loop\nwhile (true) stream = transport.catchUp(seq);\n// after: backoff on quota errors\nlet delay = 1000;\nwhile (true) {\n  try { stream = transport.catchUp(seq); break; }\n  catch (e) {\n    if (e instanceof PhotonError && e.code === \"quota\") {\n      await sleep(delay + Math.random() * delay);\n      delay = Math.min(delay * 2, 60_000);\n      continue;\n    }\n    throw e;\n  }\n}","handlingStrategy":"retry","validationCode":"// gauge catchup pressure before opening the stream\nconst gap = lastKnownSequence != null ? currentSequence - lastKnownSequence : Infinity;\nif (gap > 100_000) console.warn(\"Large recovery gap; consider staged catchup to avoid quota throttling\");","typeGuard":"function isPhotonQuotaError(e: unknown): e is PhotonError {\n  return e instanceof PhotonError && e.code === \"quota\";\n}","tryCatchPattern":"let delay = 1_000;\nfor (;;) {\n  try { return transport.catchUp(seq); }\n  catch (e) {\n    if (isPhotonQuotaError(e)) {\n      await sleep(delay + Math.random() * delay); // exponential backoff + jitter\n      delay = Math.min(delay * 2, 60_000);\n      continue;\n    }\n    throw e;\n  }\n}","preventionTips":["Never reconnect in a tight loop after failures — always back off, or you will trip the server's rate limiter","Add jitter to reconnect delays so many instances don't synchronize their retries after an outage","Keep cursors fresh (catch up frequently) so each recovery replay stays small","Track quota-error frequency per tenant to spot when limits need raising"],"tags":["grpc","rate-limit","recovery","retry"],"backgroundTag":"rate-limit-exceeded","analyzedSha":"3f1d897a7c018d76563a21c6e39c3c9b03933622","analyzedAt":"2026-09-18T08:03:59.046Z","contentChangedAt":"2026-09-18T08:03:59.046Z","schemaVersion":2},"datasetVersion":"2026-09-22T11:17:16.035Z"}