paperclipai/paperclip · error · PhotonError

network

network

Error message

Photon catch-up ended before its checkpoint barrier

What it means

A valid catch-up must end with the stream signaling completion before the checkpoint barrier is reached. If the replay stream ends (iterates to completion) without the server having sent the completion marker, and the receiver was not explicitly stopped, the catch-up is considered truncated mid-recovery and a network error is thrown rather than silently treating an incomplete replay as complete.

Solutions

1. Simply retry catchUp(): the saved cursor persists, so replay resumes from the last checkpointed sequence idempotently
2. Increase client-side and proxy idle/read timeouts so long replays are not cut off mid-stream
3. Check Photon server logs for restarts or stream-deadline kills around the failure time and raise the server's stream deadline
4. Reduce replay size (checkpoint more often, or snapshot-restore instead of replaying) so catch-up completes well inside deadline budgets

Example fix

// before: single attempt, fails on flaky stream
await receiver.catchUp(opts);
// after: idempotent retry from the persisted cursor
for (let attempt = 0; attempt < 3; attempt++) {
  try { await receiver.catchUp(opts); break; }
  catch (e) { if (e.code !== "network" || attempt === 2) throw e; await sleep(2 ** attempt * 1000); }
}

try {
  await receiver.catchUp(opts);
} catch (e) {
  if (e instanceof PhotonError && e.code === "network" && /checkpoint barrier/.test(e.message)) {
    await backoffRetry(() => receiver.catchUp(opts), { attempts: 3 }); // idempotent: resumes from persisted cursor
  } else throw e;
}

When it happens

Trigger: catchUp()'s replay loop finishes iterating without the stream having set completed=true (server closed the stream early, dropped the connection, or never sent the barrier), while this.stopped is still false.

Common situations: Transient network partition or idle-timeout between the client and Photon; a proxy/load balancer killing long-lived streams; Photon server restart or crash during replay; aggressive server-side stream deadline shorter than the replay needs.

Understand the failure class

Background: Request timed out: what client-side request timeouts mean across libraries (Request timed out, TIMED_OUT, APITimeoutError) — this error's family across 39 libraries.

Related errors

• CreateOS process stream ended without an exit status.
• Anthropic Managed Agents request failed with HTTP
• Artifact download failed: HTTP
• Artifact exceeds size limit.
• connector_refresh_failed

AI-assisted analysis of paperclipai/paperclip@3f1d897a7c (2026-09-18). Data as JSON: /api/errors/c7ff81e7de446cc0. Report an issue: GitHub.

Appendix: source

Thrown at server/src/services/photon/receiver.ts:171

          );
        // On the first connection, the server may retain only a tail of history.
        // Establish that boundary explicitly; it is never allowed after a cursor.
        sequence ??= event.sequence - 1;
        if (event.type !== "photon.ignored") {
          const occurredAt = new Date(event.occurredAt).getTime();
          if (!Number.isFinite(occurredAt))
            throw new PhotonError(
              "invalid_response",
              "Photon event timestamp is invalid",
            );
          if (occurredAt >= intakeAfter) await admit(event);
        }
        // admission must durably store or classify even irrelevant events.
        if (!shared) await this.checkpoint(event.sequence);
        sequence = event.sequence;
      }
      if (!completed && !this.stopped)
        throw new PhotonError(
          "network",
          "Photon catch-up ended before its checkpoint barrier",
        );
    } finally {
      await stream.close();
      if (this.catchUpStream === stream) this.catchUpStream = undefined;
    }
  }
  private async checkpoint(sequence: number): Promise<void> {
    await this.options.assertOwned();
    if (this.options.commitCheckpoint)
      return this.options.commitCheckpoint(sequence);
    await writePhotonCheckpoint(
      this.options.state,
      this.options.lineId,
      sequence,
    );
  }

View on GitHub (pinned to 3f1d897a7c)