apache/iceberg · error · IllegalStateException

Cannot load current offset at snapshot %d, the snapshot was

Error message

Cannot load current offset at snapshot %d, the snapshot was expired or removed

What it means

The Spark structured streaming micro-batch planner validates that the snapshot recorded in the current streaming offset still exists. If Table.currentSnapshot() returns null (the referenced snapshot was expired by retention/expireSnapshots), it throws IllegalStateException since the batch cannot be planned.

Source

Thrown at spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/source/SyncSparkMicroBatchPlanner.java:242

        startPosOfSnapOffset = -1;
        // if anyhow we are moving to next snapshot we should only scan addedFiles
        scanAllFiles = false;
      }
    }

    StreamingOffset latestStreamingOffset =
        new StreamingOffset(curSnapshot.snapshotId(), curPos, scanAllFiles);

    // if no new data arrived, then return null.
    return latestStreamingOffset.equals(startingOffset) ? null : latestStreamingOffset;
  }

  @Override
  public void stop() {}

  private void validateCurrentSnapshotExists(Snapshot snapshot, StreamingOffset currentOffset) {
    if (snapshot == null) {
      throw new IllegalStateException(
          String.format(
              Locale.ROOT,
              "Cannot load current offset at snapshot %d, the snapshot was expired or removed",
              currentOffset.snapshotId()));
    }
  }
}

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. Restart the streaming query from a checkpoint-free state or with a fresh checkpoint after verifying current snapshots exist
2. Avoid running expireSnapshots while a streaming consumer may need old snapshots; retain snapshots longer than the stream's expected downtime (history.expire-max-snapshot-age / min-snapshots-to-keep)
3. Use a new streaming query starting at current snapshot instead of the stale offset
4. Restore the table state (e.g. from rollback/backup) if the snapshot must be recoverable

Example fix

// before
spark.readStream.format("iceberg").load(table).writeStream...start(checkpoint)
// after
// after snapshot expiry invalidated the offset, start fresh:
spark.readStream.format("iceberg").option("stream-from-timestamp", restartTs).load(table)...

Snapshot snap = table.currentSnapshot();
if (snap == null || snap.snapshotId() != lastOffset.snapshotId()) {
  // offset is stale; restart the stream from a fresh offset
}

try {
  startStream(checkpoint);
} catch (IllegalStateException e) {
  if (e.getMessage().contains("the snapshot was expired or removed")) {
    deleteCheckpoint();
    startStreamFresh();
  } else throw e;
}

When it happens

Trigger: Streaming query resumes with an offset whose snapshotId points to a snapshot that has since been expired (expireSnapshots ran, or retention removed it) while the stream was stopped.

Common situations: Long-stopped streaming job restarted after expireSnapshots ran on the table; aggressive snapshot expiration in concurrent jobs; table replaced/rebuilt under the same location.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• Cannot load current offset at snapshot %d, the snapshot was
• Cannot load current offset at snapshot %d, the snapshot was
• Interrupted while polling queue
• Table refresh failed
• Queue filling failed

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/91df36aa71b7dd41. Report an issue: GitHub.