{"record":{"id":"b72d0b5e0b296cab","repo":"apache/iceberg","slug":"cannot-load-current-offset-at-snapshot-d-the-sna-b72d0b","errorCode":null,"errorMessage":"Cannot load current offset at snapshot %d, the snapshot was expired or removed","messagePattern":"Cannot load current offset at snapshot (.+?), the snapshot was expired or removed","errorType":"exception","errorClass":"IllegalStateException","httpStatus":null,"severity":"error","filePath":"spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/source/SyncSparkMicroBatchPlanner.java","lineNumber":242,"sourceCode":"        startPosOfSnapOffset = -1;\n        // if anyhow we are moving to next snapshot we should only scan addedFiles\n        scanAllFiles = false;\n      }\n    }\n\n    StreamingOffset latestStreamingOffset =\n        new StreamingOffset(curSnapshot.snapshotId(), curPos, scanAllFiles);\n\n    // if no new data arrived, then return null.\n    return latestStreamingOffset.equals(startingOffset) ? null : latestStreamingOffset;\n  }\n\n  @Override\n  public void stop() {}\n\n  private void validateCurrentSnapshotExists(Snapshot snapshot, StreamingOffset currentOffset) {\n    if (snapshot == null) {\n      throw new IllegalStateException(\n          String.format(\n              Locale.ROOT,\n              \"Cannot load current offset at snapshot %d, the snapshot was expired or removed\",\n              currentOffset.snapshotId()));\n    }\n  }\n}\n","sourceCodeStart":224,"sourceCodeEnd":250,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/source/SyncSparkMicroBatchPlanner.java#L224-L250","documentation":"During Spark structured streaming planning, validateCurrentSnapshotExists checks that the snapshot referenced by the current streaming offset still exists. If it was expired or removed (e.g. by expireSnapshots or table replacement), the planner throws IllegalStateException because it cannot compute an increment from a missing starting point.","triggerScenarios":"Streaming micro-batch planning (planFiles/latestOffset) when the snapshotId recorded in the streaming offset has been expired via expireSnapshots, the table was rewritten/replaced, or retention removed the snapshot between batches.","commonSituations":"Long-paused streaming query resumed after snapshots were expired; aggressive snapshot expiration jobs running while a stream is active; branch overwrite/replacement invalidating offsets.","solutions":["Restart the streaming query from the current/latest snapshot (new checkpoint) since the old offset is unrecoverable","Increase retention (e.g. history.expire.max-snapshot-age-ms) or pause expiration jobs while streams run","Restore the snapshot from a backup or recover table metadata if possible","Align expiration policies with streaming checkpoint intervals going forward"],"exampleFix":"// before: expiring snapshots aggressively\nALTER TABLE t EXECUTE expire_snapshots(retention_threshold => '1 h');\n// after: keep snapshots older than streaming checkpoint interval\nALTER TABLE t EXECUTE expire_snapshots(retention_threshold => '7 d');","handlingStrategy":"retry","validationCode":"Snapshot s = table.snapshot(offset.snapshotId());\nif (s == null) {\n  // offset is stale: restart stream from latest snapshot before calling planFiles\n}","typeGuard":null,"tryCatchPattern":"try {\n  batch = planner.planFiles(offset);\n} catch (IllegalStateException e) {\n  if (e.getMessage().contains(\"expired or removed\")) {\n    stream.resetLatest(); // restart from latest snapshot with fresh checkpoint\n  } else {\n    throw e;\n  }\n}","preventionTips":["Keep snapshot retention longer than the max streaming pause duration","Coordinate expire_snapshots jobs with active streaming queries","Alert on streaming checkpoints that lag behind expiration windows"],"tags":["spark","streaming","snapshot-expired"],"backgroundTag":"resource-not-found","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}