apache/iceberg · error · ValidationException

Found conflicting files that can contain records matching %s

Error message

Found conflicting files that can contain records matching %s: %s

What it means

Filter-based variant of the conflicting-files validation. validateAddedDataFiles scans data files added since the starting snapshot that match the given conflictDetectionFilter; if any exist, it throws this ValidationException listing the filter and the conflicting file locations. This prevents commits (e.g. deletes/overwrites) from being based on data that has since changed.

Source

Thrown at core/src/main/java/org/apache/iceberg/MergingSnapshotProducer.java:401

  /**
   * Validates that no files matching a filter have been added to the table since a starting
   * snapshot.
   *
   * @param base table metadata to validate
   * @param startingSnapshotId id of the snapshot current at the start of the operation
   * @param conflictDetectionFilter an expression used to find new conflicting data files
   */
  protected void validateAddedDataFiles(
      TableMetadata base,
      Long startingSnapshotId,
      Expression conflictDetectionFilter,
      Snapshot parent) {
    CloseableIterable<ManifestEntry<DataFile>> conflictEntries =
        addedDataFiles(base, startingSnapshotId, conflictDetectionFilter, null, parent);

    try (CloseableIterator<ManifestEntry<DataFile>> conflicts = conflictEntries.iterator()) {
      if (conflicts.hasNext()) {
        throw new ValidationException(
            "Found conflicting files that can contain records matching %s: %s",
            conflictDetectionFilter,
            Iterators.toString(
                Iterators.transform(conflicts, entry -> entry.file().location().toString())));
      }

    } catch (IOException e) {
      throw new UncheckedIOException(
          String.format("Failed to validate no appends matching %s", conflictDetectionFilter), e);
    }
  }

  /**
   * Returns an iterable of files matching a filter have been added to a branch since a starting
   * snapshot.
   *
   * @param base table metadata to validate
   * @param startingSnapshotId id of the snapshot current at the start of the operation

View on GitHub (pinned to 86d9c8fc54)

Solutions

Refresh the table and rebuild/retry the operation against the current snapshot
Coordinate or serialize workloads touching overlapping predicates (partition the workload to disjoint filters)
Use a lower isolation/relax validation properties (e.g. write.delete.validation-pressure or skip delete validation) only when safe
Retry pattern: catch ValidationException, reload table, re-plan, re-commit

Example fix

// before
rowDelta.commit(); // may throw ValidationException on concurrent appends
// after
try {
  rowDelta.commit();
} catch (ValidationException e) {
  Table refreshed = catalog.loadTable(id);
  refreshed.newRowDelta()... // rebuild deletes against latest snapshot
    .commit();
}

Defensive patterns

Strategy: retry

Validate before calling

// check for appends matching the filter since the planned snapshot
Snapshot since = table.snapshot(plannedBaseSnapshotId);
if (table.currentSnapshot().snapshotId() != plannedBaseSnapshotId) { /* re-validate or re-plan before commit */ }

Try / catch

try { rowDelta.commit(); } catch (ValidationException e) { if (e.getMessage().startsWith("Found conflicting files that can contain records matching")) { table.refresh(); /* rebuild deletes against latest snapshot */ } else throw e; }

Prevention

Minimize time between snapshot load and commit
Use retry-with-refresh loops for delete/overwrite commits in concurrent workloads
Design filters/partitions so concurrent writers operate on disjoint data ranges
Consider stronger coordination (external locks) for overlapping predicates

When it happens

Trigger: A delete/overwrite/RowDelta commit validates that no new data files matching its conflict-detection filter were added since its starting snapshot; a concurrent job appended rows matching the filter (e.g. same equality-delete keys or same row filter).

Common situations: Concurrent UPDATE/DELETE and INSERT on overlapping row ranges; equality-delete commits racing with appends of matching keys; stale compaction jobs validating old predicates.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/927ba87895215ca7. Report an issue: GitHub.