{"record":{"id":"91df36aa71b7dd41","repo":"apache/iceberg","slug":"cannot-load-current-offset-at-snapshot-d-the-sna-91df36","errorCode":null,"errorMessage":"Cannot load current offset at snapshot %d, the snapshot was expired or removed","messagePattern":"Cannot load current offset at snapshot (.+?), the snapshot was expired or removed","errorType":"exception","errorClass":"IllegalStateException","httpStatus":null,"severity":"error","filePath":"spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/source/SyncSparkMicroBatchPlanner.java","lineNumber":242,"sourceCode":"        startPosOfSnapOffset = -1;\n        // if anyhow we are moving to next snapshot we should only scan addedFiles\n        scanAllFiles = false;\n      }\n    }\n\n    StreamingOffset latestStreamingOffset =\n        new StreamingOffset(curSnapshot.snapshotId(), curPos, scanAllFiles);\n\n    // if no new data arrived, then return null.\n    return latestStreamingOffset.equals(startingOffset) ? null : latestStreamingOffset;\n  }\n\n  @Override\n  public void stop() {}\n\n  private void validateCurrentSnapshotExists(Snapshot snapshot, StreamingOffset currentOffset) {\n    if (snapshot == null) {\n      throw new IllegalStateException(\n          String.format(\n              Locale.ROOT,\n              \"Cannot load current offset at snapshot %d, the snapshot was expired or removed\",\n              currentOffset.snapshotId()));\n    }\n  }\n}\n","sourceCodeStart":224,"sourceCodeEnd":250,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/source/SyncSparkMicroBatchPlanner.java#L224-L250","documentation":"The Spark structured streaming micro-batch planner validates that the snapshot recorded in the current streaming offset still exists. If Table.currentSnapshot() returns null (the referenced snapshot was expired by retention/expireSnapshots), it throws IllegalStateException since the batch cannot be planned.","triggerScenarios":"Streaming query resumes with an offset whose snapshotId points to a snapshot that has since been expired (expireSnapshots ran, or retention removed it) while the stream was stopped.","commonSituations":"Long-stopped streaming job restarted after expireSnapshots ran on the table; aggressive snapshot expiration in concurrent jobs; table replaced/rebuilt under the same location.","solutions":["Restart the streaming query from a checkpoint-free state or with a fresh checkpoint after verifying current snapshots exist","Avoid running expireSnapshots while a streaming consumer may need old snapshots; retain snapshots longer than the stream's expected downtime (history.expire-max-snapshot-age / min-snapshots-to-keep)","Use a new streaming query starting at current snapshot instead of the stale offset","Restore the table state (e.g. from rollback/backup) if the snapshot must be recoverable"],"exampleFix":"// before\nspark.readStream.format(\"iceberg\").load(table).writeStream...start(checkpoint)\n// after\n// after snapshot expiry invalidated the offset, start fresh:\nspark.readStream.format(\"iceberg\").option(\"stream-from-timestamp\", restartTs).load(table)...","handlingStrategy":"retry","validationCode":"Snapshot snap = table.currentSnapshot();\nif (snap == null || snap.snapshotId() != lastOffset.snapshotId()) {\n  // offset is stale; restart the stream from a fresh offset\n}","typeGuard":null,"tryCatchPattern":"try {\n  startStream(checkpoint);\n} catch (IllegalStateException e) {\n  if (e.getMessage().contains(\"the snapshot was expired or removed\")) {\n    deleteCheckpoint();\n    startStreamFresh();\n  } else throw e;\n}","preventionTips":["Keep snapshot retention longer than maximum expected stream downtime","Don't run expireSnapshots concurrently with streaming readers","Monitor for stale checkpoints after table maintenance"],"tags":["spark","streaming","snapshot-expired"],"backgroundTag":"invalid-state-transition","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}