paperclipai/paperclip · critical · PhotonError

history_gap

history_gap

Error message

Photon receiver checkpoint is missing; operator recovery is required

What it means

PhotonReceiver.catchUp reads the persisted 'checkpoint' from PhotonState. If there is no checkpoint but a 'receiver-initialized' marker exists, the durable cursor was lost while the receiver state says initialization happened — an unrecoverable-looking history hole. It throws PhotonError with code 'history_gap' because events after the lost cursor could be missed; automatic replay cannot know where to resume safely.

Solutions

Follow the operator recovery procedure: clear the 'receiver-initialized' marker (or re-initialize state) so the next catchUp can establish a fresh boundary cursor, accepting a tail-only replay.
Restore the checkpoint key from a backup taken at or after the last processed sequence.
If commitCheckpoint is configured, verify it persisted the cursor transactionally with the lease fence; fix any path that writes 'receiver-initialized' without a checkpoint.
Recreate the receiver state (state store reset + fresh initialization) if missed events are acceptable or re-fetchable.

Example fix

// before (operator shell)
state.del('checkpoint');
// after — clear both keys together so state is consistent
state.del('checkpoint');
state.del('receiver-initialized');

Defensive patterns

Strategy: try-catch

Validate before calling

export async function isReceiverStateConsistent(state: PhotonState): Promise<boolean> {
  const checkpoint = await state.read('checkpoint');
  const initialized = await state.read('receiver-initialized');
  return !(initialized && !checkpoint);
}

Try / catch

try {
  await receiver.catchUp();
} catch (err) {
  if (err instanceof PhotonError && err.code === 'history_gap' &&
      err.message.includes('checkpoint is missing')) {
    await operatorRecovery(state); // clear receiver-initialized, re-init boundary
    return;
  }
  throw err;
}

Prevention

Always write the checkpoint and the receiver-initialized marker atomically (use commitCheckpoint)
Never delete the checkpoint key alone; clear both state keys together
Back up PhotonState as a whole, not individual keys
Alert on state-store restore operations so recovery can be planned

When it happens

Trigger: catchUp() runs when state.read('checkpoint') returns undefined while state.read('receiver-initialized') is truthy — e.g. the checkpoint key was deleted or corrupted, state storage was partially restored from a backup, or a manual DB/state edit removed the cursor but kept the init flag.

Common situations: Restoring PhotonState storage from an older snapshot that predates checkpoint writes; operator manually clearing keys; a state-store migration bug dropping one key but not another; running two receivers against one state store where one wiped it.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

AI-assisted analysis of paperclipai/paperclip@3f1d897a7c (2026-09-18). Data as JSON: /api/errors/46417ea3b9574a43. Report an issue: GitHub.

Appendix: source

Thrown at server/src/services/photon/receiver.ts:93

        this.requested = false;
        await this.catchUp();
      }
    })()
      .catch(async (error) => {
        if (!this.stopped) await this.options.failure(error);
      })
      .finally(() => {
        this.running = undefined;
      });
  }
  /** Exposed for deterministic recovery tests, never an external route. */
  async catchUp(): Promise<void> {
    const { state, lineId, client, assertOwned, admit, intakeAfter } =
      this.options;
    await assertOwned();
    const original = await state.read<Checkpoint>("checkpoint");
    if (!original && (await state.read("receiver-initialized")))
      throw new PhotonError(
        "history_gap",
        "Photon receiver checkpoint is missing; operator recovery is required",
      );
    if (
      original &&
      (original.schema !== 1 ||
        original.lineId !== lineId ||
        !Number.isSafeInteger(original.sequence) ||
        original.sequence < 0)
    )
      throw new PhotonError(
        "history_gap",
        "Photon checkpoint is invalid; operator recovery is required",
      );
    const shared = this.options.allocation === "shared";
    let sequence = original?.sequence;
    let batchEvents = 0;
    const stream =

View on GitHub (pinned to 3f1d897a7c)