{"record":{"id":"265d1b21f76b9cb6","repo":"apache/iceberg","slug":"found-s-operation-cannot-support-incremental-dat","errorCode":null,"errorMessage":"Found %s operation, cannot support incremental data in snapshots (%s, %s]","messagePattern":"Found (.+?) operation, cannot support incremental data in snapshots \\((.+?), (.+?)\\]","errorType":"exception","errorClass":"UnsupportedOperationException","httpStatus":null,"severity":"error","filePath":"core/src/main/java/org/apache/iceberg/IncrementalDataTableScan.java","lineNumber":139,"sourceCode":"    return manifestGroup.planFiles();\n  }\n\n  @Override\n  @SuppressWarnings(\"checkstyle:HiddenField\")\n  protected TableScan newRefinedScan(Table table, Schema schema, TableScanContext context) {\n    return new IncrementalDataTableScan(table, schema, context);\n  }\n\n  private static List<Snapshot> snapshotsWithin(\n      Table table, long fromSnapshotId, long toSnapshotId) {\n    List<Snapshot> snapshots = Lists.newArrayList();\n    for (Snapshot snapshot :\n        SnapshotUtil.ancestorsBetween(toSnapshotId, fromSnapshotId, table::snapshot)) {\n      // for now, incremental scan supports only appends\n      if (snapshot.operation().equals(DataOperations.APPEND)) {\n        snapshots.add(snapshot);\n      } else if (snapshot.operation().equals(DataOperations.OVERWRITE)) {\n        throw new UnsupportedOperationException(\n            String.format(\n                \"Found %s operation, cannot support incremental data in snapshots (%s, %s]\",\n                DataOperations.OVERWRITE, fromSnapshotId, toSnapshotId));\n      }\n    }\n    return snapshots;\n  }\n\n  private void validateSnapshotIdsRefinement(long newFromSnapshotId, long newToSnapshotId) {\n    Set<Long> snapshotIdsRange =\n        Sets.newHashSet(\n            SnapshotUtil.ancestorIdsBetween(\n                context().toSnapshotId(), context().fromSnapshotId(), table()::snapshot));\n    // since snapshotIdsBetween return ids in range (fromSnapshotId, toSnapshotId]\n    snapshotIdsRange.add(context().fromSnapshotId());\n    Preconditions.checkArgument(\n        snapshotIdsRange.contains(newFromSnapshotId),\n        \"from snapshot id %s not in existing snapshot ids range (%s, %s]\",","sourceCodeStart":121,"sourceCodeEnd":157,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/core/src/main/java/org/apache/iceberg/IncrementalDataTableScan.java#L121-L157","documentation":"Incremental scans currently support only APPEND snapshots: any overwrite rewrites existing rows, which a difference-of-manifests incremental read cannot represent safely. snapshotsWithin walks the ancestor chain between the range endpoints and throws UnsupportedOperationException if it encounters an OVERWRITE snapshot, reporting the range.","triggerScenarios":"Calling table.incrementalScan(fromId, toId) (or .planFiles()/tasks()) where any ancestor snapshot between fromId (exclusive) and toId (inclusive) has operation 'overwrite'.","commonSituations":"Tables that received overwrite/compaction-with-overwrite (e.g. MERGE OVERWRITE, dynamic partition overwrite via a non-appending writer) inside the requested range; users scanning ranges that span maintenance jobs.","solutions":["Choose a fromSnapshotId after the last overwrite snapshot so the range contains only appends.","Use the changelog scan (IncrementalChangelogScan) or table.scan() with row-level filters if you must read a range containing overwrites.","Audit snapshot operations in the range via SnapshotUtil.ancestorsBetween to find the offending overwrite snapshot.","Rewrite the workflow that produced the overwrite to use append-only semantics (e.g. copy-on-write delete+append) if incremental append scans are a hard requirement."],"exampleFix":"// before\nTableScan scan = table.incrementalScan(earliestId, currentId); // may include overwrites\n// after\nlong safeFrom = SnapshotUtil.ancestorsBetween(currentId, null, table::snapshot).stream()\n    .filter(id -> DataOperations.APPEND.equals(table.snapshot(id).operation()))\n    .filter(id -> id > earliestId)\n    .findFirst().orElse(earliestId);\nTableScan scan = table.incrementalScan(safeFrom, currentId); // appends only","handlingStrategy":"try-catch","validationCode":"import org.apache.iceberg.SnapshotUtil;\nimport org.apache.iceberg.DataOperations;\nboolean rangeAppendOnly(long fromId, long toId, Table table) {\n  return SnapshotUtil.ancestorsBetween(toId, fromId, table::snapshot).stream()\n      .allMatch(id -> DataOperations.APPEND.equals(table.snapshot(id).operation()));\n}\n// check before calling incrementalScan; otherwise use changelog or a full scan","typeGuard":null,"tryCatchPattern":"try {\n  TableScan scan = table.incrementalScan(fromId, toId);\n  scan.planFiles();\n} catch (UnsupportedOperationException e) {\n  // range contains an overwrite; fall back to changelog or full scan\n  TableScan scan = table.incrementalChangelogScan().fromSnapshotId(fromId);\n}","preventionTips":["Check snapshot operations in a range before incremental scans; keep maintenance overwrites outside scanned ranges.","Prefer changelog scans when row-level operations matter.","Snapshot audits with SnapshotUtil.ancestorsBetween become part of pipeline validation."],"tags":["incremental-scan","overwrite","data-operation","unsupported"],"backgroundTag":"unsupported-operation","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}