{"record":{"id":"46417ea3b9574a43","repo":"paperclipai/paperclip","slug":"history-gap","errorCode":"history_gap","errorMessage":"Photon receiver checkpoint is missing; operator recovery is required","messagePattern":"Photon receiver checkpoint is missing; operator recovery is required","errorType":"error_code","errorClass":"PhotonError","httpStatus":null,"severity":"critical","filePath":"server/src/services/photon/receiver.ts","lineNumber":93,"sourceCode":"        this.requested = false;\n        await this.catchUp();\n      }\n    })()\n      .catch(async (error) => {\n        if (!this.stopped) await this.options.failure(error);\n      })\n      .finally(() => {\n        this.running = undefined;\n      });\n  }\n  /** Exposed for deterministic recovery tests, never an external route. */\n  async catchUp(): Promise<void> {\n    const { state, lineId, client, assertOwned, admit, intakeAfter } =\n      this.options;\n    await assertOwned();\n    const original = await state.read<Checkpoint>(\"checkpoint\");\n    if (!original && (await state.read(\"receiver-initialized\")))\n      throw new PhotonError(\n        \"history_gap\",\n        \"Photon receiver checkpoint is missing; operator recovery is required\",\n      );\n    if (\n      original &&\n      (original.schema !== 1 ||\n        original.lineId !== lineId ||\n        !Number.isSafeInteger(original.sequence) ||\n        original.sequence < 0)\n    )\n      throw new PhotonError(\n        \"history_gap\",\n        \"Photon checkpoint is invalid; operator recovery is required\",\n      );\n    const shared = this.options.allocation === \"shared\";\n    let sequence = original?.sequence;\n    let batchEvents = 0;\n    const stream =","sourceCodeStart":75,"sourceCodeEnd":111,"githubUrl":"https://github.com/paperclipai/paperclip/blob/3f1d897a7c018d76563a21c6e39c3c9b03933622/server/src/services/photon/receiver.ts#L75-L111","documentation":"PhotonReceiver.catchUp reads the persisted 'checkpoint' from PhotonState. If there is no checkpoint but a 'receiver-initialized' marker exists, the durable cursor was lost while the receiver state says initialization happened — an unrecoverable-looking history hole. It throws PhotonError with code 'history_gap' because events after the lost cursor could be missed; automatic replay cannot know where to resume safely.","triggerScenarios":"catchUp() runs when state.read('checkpoint') returns undefined while state.read('receiver-initialized') is truthy — e.g. the checkpoint key was deleted or corrupted, state storage was partially restored from a backup, or a manual DB/state edit removed the cursor but kept the init flag.","commonSituations":"Restoring PhotonState storage from an older snapshot that predates checkpoint writes; operator manually clearing keys; a state-store migration bug dropping one key but not another; running two receivers against one state store where one wiped it.","solutions":["Follow the operator recovery procedure: clear the 'receiver-initialized' marker (or re-initialize state) so the next catchUp can establish a fresh boundary cursor, accepting a tail-only replay.","Restore the checkpoint key from a backup taken at or after the last processed sequence.","If commitCheckpoint is configured, verify it persisted the cursor transactionally with the lease fence; fix any path that writes 'receiver-initialized' without a checkpoint.","Recreate the receiver state (state store reset + fresh initialization) if missed events are acceptable or re-fetchable."],"exampleFix":"// before (operator shell)\nstate.del('checkpoint');\n// after — clear both keys together so state is consistent\nstate.del('checkpoint');\nstate.del('receiver-initialized');","handlingStrategy":"try-catch","validationCode":"export async function isReceiverStateConsistent(state: PhotonState): Promise<boolean> {\n  const checkpoint = await state.read('checkpoint');\n  const initialized = await state.read('receiver-initialized');\n  return !(initialized && !checkpoint);\n}","typeGuard":null,"tryCatchPattern":"try {\n  await receiver.catchUp();\n} catch (err) {\n  if (err instanceof PhotonError && err.code === 'history_gap' &&\n      err.message.includes('checkpoint is missing')) {\n    await operatorRecovery(state); // clear receiver-initialized, re-init boundary\n    return;\n  }\n  throw err;\n}","preventionTips":["Always write the checkpoint and the receiver-initialized marker atomically (use commitCheckpoint)","Never delete the checkpoint key alone; clear both state keys together","Back up PhotonState as a whole, not individual keys","Alert on state-store restore operations so recovery can be planned"],"tags":["photon","state-corruption","recovery","history-gap"],"backgroundTag":"internal-invariant-violation","analyzedSha":"3f1d897a7c018d76563a21c6e39c3c9b03933622","analyzedAt":"2026-09-18T08:03:59.046Z","contentChangedAt":"2026-09-18T08:03:59.046Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}