{"record":{"id":"c7ff81e7de446cc0","repo":"paperclipai/paperclip","slug":"network-photon-catch-up-ended-before-its-checkpoint-barrier","errorCode":"network","errorMessage":"Photon catch-up ended before its checkpoint barrier","messagePattern":"Photon catch-up ended before its checkpoint barrier","errorType":"error_code","errorClass":"PhotonError","httpStatus":null,"severity":"error","filePath":"server/src/services/photon/receiver.ts","lineNumber":171,"sourceCode":"          );\n        // On the first connection, the server may retain only a tail of history.\n        // Establish that boundary explicitly; it is never allowed after a cursor.\n        sequence ??= event.sequence - 1;\n        if (event.type !== \"photon.ignored\") {\n          const occurredAt = new Date(event.occurredAt).getTime();\n          if (!Number.isFinite(occurredAt))\n            throw new PhotonError(\n              \"invalid_response\",\n              \"Photon event timestamp is invalid\",\n            );\n          if (occurredAt >= intakeAfter) await admit(event);\n        }\n        // admission must durably store or classify even irrelevant events.\n        if (!shared) await this.checkpoint(event.sequence);\n        sequence = event.sequence;\n      }\n      if (!completed && !this.stopped)\n        throw new PhotonError(\n          \"network\",\n          \"Photon catch-up ended before its checkpoint barrier\",\n        );\n    } finally {\n      await stream.close();\n      if (this.catchUpStream === stream) this.catchUpStream = undefined;\n    }\n  }\n  private async checkpoint(sequence: number): Promise<void> {\n    await this.options.assertOwned();\n    if (this.options.commitCheckpoint)\n      return this.options.commitCheckpoint(sequence);\n    await writePhotonCheckpoint(\n      this.options.state,\n      this.options.lineId,\n      sequence,\n    );\n  }","sourceCodeStart":153,"sourceCodeEnd":189,"githubUrl":"https://github.com/paperclipai/paperclip/blob/3f1d897a7c018d76563a21c6e39c3c9b03933622/server/src/services/photon/receiver.ts#L153-L189","documentation":"A valid catch-up must end with the stream signaling completion before the checkpoint barrier is reached. If the replay stream ends (iterates to completion) without the server having sent the completion marker, and the receiver was not explicitly stopped, the catch-up is considered truncated mid-recovery and a network error is thrown rather than silently treating an incomplete replay as complete.","triggerScenarios":"catchUp()'s replay loop finishes iterating without the stream having set completed=true (server closed the stream early, dropped the connection, or never sent the barrier), while this.stopped is still false.","commonSituations":"Transient network partition or idle-timeout between the client and Photon; a proxy/load balancer killing long-lived streams; Photon server restart or crash during replay; aggressive server-side stream deadline shorter than the replay needs.","solutions":["Simply retry catchUp(): the saved cursor persists, so replay resumes from the last checkpointed sequence idempotently","Increase client-side and proxy idle/read timeouts so long replays are not cut off mid-stream","Check Photon server logs for restarts or stream-deadline kills around the failure time and raise the server's stream deadline","Reduce replay size (checkpoint more often, or snapshot-restore instead of replaying) so catch-up completes well inside deadline budgets"],"exampleFix":"// before: single attempt, fails on flaky stream\nawait receiver.catchUp(opts);\n// after: idempotent retry from the persisted cursor\nfor (let attempt = 0; attempt < 3; attempt++) {\n  try { await receiver.catchUp(opts); break; }\n  catch (e) { if (e.code !== \"network\" || attempt === 2) throw e; await sleep(2 ** attempt * 1000); }\n}","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  await receiver.catchUp(opts);\n} catch (e) {\n  if (e instanceof PhotonError && e.code === \"network\" && /checkpoint barrier/.test(e.message)) {\n    await backoffRetry(() => receiver.catchUp(opts), { attempts: 3 }); // idempotent: resumes from persisted cursor\n  } else throw e;\n}","preventionTips":["Set client idle/read timeouts comfortably above your worst-case replay duration","Configure proxies/LBs to not kill long-lived streams mid-replay","Checkpoint frequently to keep each catch-up window short","Retry logic is safe here because checkpoints make catch-up idempotent"],"tags":["photon","network","stream","replay"],"backgroundTag":"request-timeout","analyzedSha":"3f1d897a7c018d76563a21c6e39c3c9b03933622","analyzedAt":"2026-09-18T08:03:59.046Z","contentChangedAt":"2026-09-18T08:03:59.046Z","schemaVersion":2},"datasetVersion":"2026-09-22T01:17:13.364Z"}