apache/iceberg · error · IllegalStateException

Cannot load current offset at snapshot %d, the snapshot was

Error message

Cannot load current offset at snapshot %d, the snapshot was expired or removed

What it means

During Spark structured streaming planning, validateCurrentSnapshotExists checks that the snapshot referenced by the current streaming offset still exists. If it was expired or removed (e.g. by expireSnapshots or table replacement), the planner throws IllegalStateException because it cannot compute an increment from a missing starting point.

Source

Thrown at spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/source/SyncSparkMicroBatchPlanner.java:242

        startPosOfSnapOffset = -1;
        // if anyhow we are moving to next snapshot we should only scan addedFiles
        scanAllFiles = false;
      }
    }

    StreamingOffset latestStreamingOffset =
        new StreamingOffset(curSnapshot.snapshotId(), curPos, scanAllFiles);

    // if no new data arrived, then return null.
    return latestStreamingOffset.equals(startingOffset) ? null : latestStreamingOffset;
  }

  @Override
  public void stop() {}

  private void validateCurrentSnapshotExists(Snapshot snapshot, StreamingOffset currentOffset) {
    if (snapshot == null) {
      throw new IllegalStateException(
          String.format(
              Locale.ROOT,
              "Cannot load current offset at snapshot %d, the snapshot was expired or removed",
              currentOffset.snapshotId()));
    }
  }
}

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. Restart the streaming query from the current/latest snapshot (new checkpoint) since the old offset is unrecoverable
2. Increase retention (e.g. history.expire.max-snapshot-age-ms) or pause expiration jobs while streams run
3. Restore the snapshot from a backup or recover table metadata if possible
4. Align expiration policies with streaming checkpoint intervals going forward

Example fix

// before: expiring snapshots aggressively
ALTER TABLE t EXECUTE expire_snapshots(retention_threshold => '1 h');
// after: keep snapshots older than streaming checkpoint interval
ALTER TABLE t EXECUTE expire_snapshots(retention_threshold => '7 d');

Snapshot s = table.snapshot(offset.snapshotId());
if (s == null) {
  // offset is stale: restart stream from latest snapshot before calling planFiles
}

try {
  batch = planner.planFiles(offset);
} catch (IllegalStateException e) {
  if (e.getMessage().contains("expired or removed")) {
    stream.resetLatest(); // restart from latest snapshot with fresh checkpoint
  } else {
    throw e;
  }
}

When it happens

Trigger: Streaming micro-batch planning (planFiles/latestOffset) when the snapshotId recorded in the streaming offset has been expired via expireSnapshots, the table was rewritten/replaced, or retention removed the snapshot between batches.

Common situations: Long-paused streaming query resumed after snapshots were expired; aggressive snapshot expiration jobs running while a stream is active; branch overwrite/replacement invalidating offsets.

Understand the failure class

Background: 'Could not be found', 'does not exist', 'not found in database': the resource-not-found family when an ID, slug, key, or URI lookup comes back empty — this error's family across 20 libraries.

Related errors

• Cannot load current offset at snapshot %d, the snapshot was
• Cannot load current offset at snapshot %d, the snapshot was
• Interrupted while polling queue
• Table refresh failed
• Queue filling failed

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/b72d0b5e0b296cab. Report an issue: GitHub.