apache/iceberg · error · ValidationException

Found conflicting deleted files that can contain records mat

Error message

Found conflicting deleted files that can contain records matching %s: %s

What it means

ValidationException thrown during commit when another concurrent snapshot deleted data files that could contain records matching the given filter since the starting snapshot. Iceberg aborts the commit because proceeding could lose or corrupt data affected by those concurrent deletes. This is a genuine optimistic-concurrency conflict, not an I/O bug.

Source

Thrown at core/src/main/java/org/apache/iceberg/MergingSnapshotProducer.java:687

  }

  /**
   * Validates that no files matching a filter have been deleted from the table since a starting
   * snapshot.
   *
   * @param base table metadata to validate
   * @param startingSnapshotId id of the snapshot current at the start of the operation
   * @param dataFilter an expression used to find deleted data files
   * @param parent ending snapshot on the branch being validated
   */
  protected void validateDeletedDataFiles(
      TableMetadata base, Long startingSnapshotId, Expression dataFilter, Snapshot parent) {
    CloseableIterable<ManifestEntry<DataFile>> conflictEntries =
        deletedDataFiles(base, startingSnapshotId, dataFilter, null, parent);

    try (CloseableIterator<ManifestEntry<DataFile>> conflicts = conflictEntries.iterator()) {
      if (conflicts.hasNext()) {
        throw new ValidationException(
            "Found conflicting deleted files that can contain records matching %s: %s",
            dataFilter,
            Iterators.toString(
                Iterators.transform(conflicts, entry -> entry.file().location().toString())));
      }

    } catch (IOException e) {
      throw new UncheckedIOException(
          String.format("Failed to validate no deleted data files matching %s", dataFilter), e);
    }
  }

  /**
   * Validates that no files matching a filter have been deleted from the table since a starting
   * snapshot.
   *
   * @param base table metadata to validate
   * @param startingSnapshotId id of the snapshot current at the start of the operation

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. Refresh the table (table.refresh()), rebase the operation on the current snapshot, and re-apply the change.
2. Retry the whole operation after refresh; only one writer's change will win per snapshot.
3. Reduce conflict scope by narrowing the data filter/partition so concurrent deletes in other partitions don't clash.
4. Coordinate writers (external locking or single-writer pattern) for heavily contended tables.

Example fix

// before
table.newOverwrite().overwriteByRowFilter(filter).validateNoDeletedFiles(startSnapshotId).commit(); // may throw
// after
table.refresh();
OverwriteFiles op = table.newOverwrite().overwriteByRowFilter(filter);
op.validateNoDeletedFiles(table.currentSnapshot().snapshotId()); // validate against latest snapshot
op.commit();

table.refresh(); // rebase on latest snapshot before committing
Snapshot current = table.currentSnapshot();

try {
  operation.commit();
} catch (ValidationException e) {
  table.refresh();
  // rebuild and re-apply the operation on the current snapshot, then retry
}

When it happens

Trigger: Committing an operation (e.g., overwrite/delete) that calls validateNoDeletedFiles with a data filter while a concurrent committed snapshot deleted data files matching that filter on the same branch.

Common situations: Two writers overwriting the same partition concurrently; a compaction/expiry job deleting files while another job rewrites them; long-running streaming commits overlapping with delete operations.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• Found conflicting deleted files that can apply to records ma
• Failed to commit
• Failed to validate no deleted data files matching %s
• Cannot commit, missing data files: %s
• Cannot commit changes based on stale table metadata

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/3603aa261f1c7be6. Report an issue: GitHub.