{"record":{"id":"bac2aba41c2a9ae0","repo":"apache/iceberg","slug":"cannot-load-current-offset-at-snapshot-d-the-sna-bac2ab","errorCode":null,"errorMessage":"Cannot load current offset at snapshot %d, the snapshot was expired or removed","messagePattern":"Cannot load current offset at snapshot (.+?), the snapshot was expired or removed","errorType":"exception","errorClass":"IllegalStateException","httpStatus":null,"severity":"error","filePath":"spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/source/SyncSparkMicroBatchPlanner.java","lineNumber":242,"sourceCode":"        startPosOfSnapOffset = -1;\n        // if anyhow we are moving to next snapshot we should only scan addedFiles\n        scanAllFiles = false;\n      }\n    }\n\n    StreamingOffset latestStreamingOffset =\n        new StreamingOffset(curSnapshot.snapshotId(), curPos, scanAllFiles);\n\n    // if no new data arrived, then return null.\n    return latestStreamingOffset.equals(startingOffset) ? null : latestStreamingOffset;\n  }\n\n  @Override\n  public void stop() {}\n\n  private void validateCurrentSnapshotExists(Snapshot snapshot, StreamingOffset currentOffset) {\n    if (snapshot == null) {\n      throw new IllegalStateException(\n          String.format(\n              Locale.ROOT,\n              \"Cannot load current offset at snapshot %d, the snapshot was expired or removed\",\n              currentOffset.snapshotId()));\n    }\n  }\n}\n","sourceCodeStart":224,"sourceCodeEnd":250,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/source/SyncSparkMicroBatchPlanner.java#L224-L250","documentation":"SyncSparkMicroBatchPlanner.validateCurrentSnapshotExists verifies that the snapshot referenced by the current streaming offset still exists. If the table's currentSnapshot lookup for currentOffset.snapshotId() returns null, the snapshot was expired or removed (e.g. by expireSnapshots or retention cleanup) and the micro-batch cannot be planned, so an IllegalStateException is thrown.","triggerScenarios":"Spark structured streaming over an Iceberg table where the snapshotId recorded in the streaming offset no longer exists — typically after expireSnapshots() ran with retention shorter than the stream's lag, or the table was rewritten/recreated while the stream was paused. Called from planFiles and latestOffset.","commonSituations":"Long streaming downtime combined with aggressive snapshot expiration (small history.expire.max-snapshot-age); scheduled maintenance jobs expiring snapshots while a stream consumes; failure to coordinate table retention settings with streaming checkpoint lifetimes.","solutions":["Restart the stream with a fresh checkpoint (or reset the streaming offset) so it resumes from the current snapshot.","Increase history.expire.max-snapshot-age and history.expire.min-snapshots-to-keep so snapshots outlive maximum stream downtime.","Coordinate expireSnapshots scheduling with streaming jobs; never expire snapshots referenced by active checkpoints.","If historical replay is not required, recreate the streaming query from the table's latest snapshot."],"exampleFix":"// before\nALTER TABLE t SET TBLPROPERTIES ('history.expire.max-snapshot-age'='3600'); // stream offline longer than 1h\n// after\nALTER TABLE t SET TBLPROPERTIES ('history.expire.max-snapshot-age'='604800', 'history.expire.min-snapshots-to-keep'='100');","handlingStrategy":"validation","validationCode":"Snapshot snap = table.snapshot(offset.snapshotId());\nif (snap == null) {\n  // snapshot expired: reset checkpoint or fail fast with an operator-visible message\n}","typeGuard":"static boolean snapshotAvailable(Table table, StreamingOffset offset) {\n  return table.snapshot(offset.snapshotId()) != null;\n}","tryCatchPattern":"try {\n  planner.planFiles(offset);\n} catch (IllegalStateException e) {\n  if (e.getMessage().contains(\"the snapshot was expired or removed\")) {\n    // restart from the latest snapshot with a new checkpoint and alert operators\n  } else throw e;\n}","preventionTips":["Set history.expire.max-snapshot-age longer than worst-case stream downtime.","Pause expireSnapshots jobs while streaming queries are offline or lagging.","Monitor snapshot count/age against streaming checkpoint lag.","Test failover: expire snapshots while a stream is stopped, then restart and verify."],"tags":["spark-streaming","snapshot-expired","iceberg","streaming-offset"],"backgroundTag":"entity-not-found","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}