{"record":{"id":"d830d1bfb4a882cf","repo":"apache/iceberg","slug":"cannot-load-current-offset-at-snapshot-d-the-sna","errorCode":null,"errorMessage":"Cannot load current offset at snapshot %d, the snapshot was expired or removed","messagePattern":"Cannot load current offset at snapshot (.+?), the snapshot was expired or removed","errorType":"exception","errorClass":"IllegalStateException","httpStatus":null,"severity":"critical","filePath":"spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SyncSparkMicroBatchPlanner.java","lineNumber":242,"sourceCode":"        startPosOfSnapOffset = -1;\n        // if anyhow we are moving to next snapshot we should only scan addedFiles\n        scanAllFiles = false;\n      }\n    }\n\n    StreamingOffset latestStreamingOffset =\n        new StreamingOffset(curSnapshot.snapshotId(), curPos, scanAllFiles);\n\n    // if no new data arrived, then return null.\n    return latestStreamingOffset.equals(startingOffset) ? null : latestStreamingOffset;\n  }\n\n  @Override\n  public void stop() {}\n\n  private void validateCurrentSnapshotExists(Snapshot snapshot, StreamingOffset currentOffset) {\n    if (snapshot == null) {\n      throw new IllegalStateException(\n          String.format(\n              Locale.ROOT,\n              \"Cannot load current offset at snapshot %d, the snapshot was expired or removed\",\n              currentOffset.snapshotId()));\n    }\n  }\n}\n","sourceCodeStart":224,"sourceCodeEnd":250,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SyncSparkMicroBatchPlanner.java#L224-L250","documentation":"Spark streaming reads track progress via StreamingOffsets that reference a snapshot ID. When resuming or planning the next micro-batch, the stored snapshot must still exist; if table snapshot expiration removed it, validateCurrentSnapshotExists throws IllegalStateException so the stream fails instead of silently skipping data.","triggerScenarios":"A running Spark structured streaming query resumes from a checkpoint whose offset references snapshot N, but snapshot N was expired by expireSnapshots, retention policies, or a concurrent table maintenance job.","commonSituations":"expireSnapshots run (manually or via a scheduled cleanup) with a retention shorter than the stream's checkpoint age; shared table where another team expires snapshots; long-paused stream resumed after snapshot cleanup.","solutions":["Restore the stream from a checkpoint taken before the referenced snapshot was expired, or start a new stream with a fresh checkpoint (accepting a new starting snapshot)","Pause snapshot expiration (increase retention / stop expireSnapshots) while streams are active; keep min-snapshot-retention above max stream pause time","If data loss from restarting is unacceptable, recover the expired snapshot from backups/metadata and reload the table state","Set expireSnapshots' older-than based on the oldest streaming checkpoint's snapshot id before expiring"],"exampleFix":"// before\nspark.sql(\"CALL catalog.system.expire_snapshots('db.t', TIMESTAMP '2026-08-01 00:00:00')\");\n// after\n// keep at least 7 days so active streams' offsets remain valid\nspark.sql(\"CALL catalog.system.expire_snapshots('db.t', TIMESTAMP '2026-09-04 00:00:00', map('streaming-max-snapshot-age-ms','604800000'))\");","handlingStrategy":"try-catch","validationCode":"// before resuming a stream, verify the checkpoint's snapshot still exists\nSnapshot s = table.snapshot(currentOffset.snapshotId());\nif (s == null) {\n  throw new IllegalStateException(\"Checkpoint snapshot \" + currentOffset.snapshotId()\n      + \" expired; restart stream from a valid snapshot\");\n}","typeGuard":null,"tryCatchPattern":"try {\n  streamQuery.processAllAvailable();\n} catch (IllegalStateException e) {\n  if (e.getMessage().contains(\"the snapshot was expired or removed\")) {\n    // restart stream with fresh checkpoint or restore the snapshot\n  } else { throw e; }\n}","preventionTips":["Keep expireSnapshots retention longer than any stream pause","Check oldest streaming checkpoint age before expiring snapshots","Coordinate table maintenance with owners of streaming jobs","Alert on streams that remain stopped beyond snapshot retention"],"tags":["spark","streaming","snapshot-expired","checkpoint"],"backgroundTag":"resource-not-found","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}