apache/iceberg · error · ValidationException

Found conflicting files that can contain records matching pa

Error message

Found conflicting files that can contain records matching partitions %s: %s

What it means

A ValidationException raised by validateAddedDataFiles (partition-based conflict check). Before committing, the producer scans manifests added since the starting snapshot for data files in the conflicting partitions; if any exist, this concurrent-append conflict is reported with the partition set and the offending file locations.

Source

Thrown at core/src/main/java/org/apache/iceberg/MergingSnapshotProducer.java:370

  }

  /**
   * Validates that no files matching given partitions have been added to the table since a starting
   * snapshot.
   *
   * @param base table metadata to validate
   * @param startingSnapshotId id of the snapshot current at the start of the operation
   * @param partitionSet a set of partitions to filter new conflicting data files
   * @param parent ending snapshot on the lineage being validated
   */
  protected void validateAddedDataFiles(
      TableMetadata base, Long startingSnapshotId, PartitionSet partitionSet, Snapshot parent) {
    CloseableIterable<ManifestEntry<DataFile>> conflictEntries =
        addedDataFiles(base, startingSnapshotId, null, partitionSet, parent);

    try (CloseableIterator<ManifestEntry<DataFile>> conflicts = conflictEntries.iterator()) {
      if (conflicts.hasNext()) {
        throw new ValidationException(
            "Found conflicting files that can contain records matching partitions %s: %s",
            partitionSet,
            Iterators.toString(
                Iterators.transform(conflicts, entry -> entry.file().location().toString())));
      }

    } catch (IOException e) {
      throw new UncheckedIOException(
          String.format("Failed to validate no appends matching %s", partitionSet), e);
    }
  }

  /**
   * Validates that no files matching a filter have been added to the table since a starting
   * snapshot.
   *
   * @param base table metadata to validate
   * @param startingSnapshotId id of the snapshot current at the start of the operation

View on GitHub (pinned to 86d9c8fc54)

Solutions

Rerun the failed operation against the latest snapshot so validation passes
Serialize writers per partition (e.g. use a lock, or write to distinct partitions)
Retry with Refresh: reload the table and rebuild the producer/plan from current metadata
Enable a different isolation level if the workload allows weaker validation (e.g. disable delete validation properties when safe)

Example fix

// before: stale producer fails validation
table.newRowDelta()... // built long ago
.commit();
// after: refresh table before rebuilding and committing
Table refreshed = catalog.loadTable(tableIdentifier);
refreshed.newRowDelta()... // rebuild from refreshed snapshot
.commit();

Defensive patterns

Strategy: retry

Validate before calling

// detect conflicting appends before commit
Snapshot current = table.currentSnapshot();
if (!current.snapshotId().equals(plannedBaseSnapshotId)) { /* refresh and re-plan: another writer appended */ }

Try / catch

try { producer.commit(); } catch (ValidationException e) { if (e.getMessage().startsWith("Found conflicting files")) { table.refresh(); /* rebuild producer */ } else throw e; }

Prevention

Rebuild producers from a freshly loaded table immediately before commit
Avoid long gaps between scan planning and commit
Partition workloads so writers touch disjoint partitions
Use table-level coordination (locks or catalog-level serialization) for hot partitions

When it happens

Trigger: Two writers append data files to the same partitions concurrently (or sequentially after the snapshot this producer was based on) while both validate no conflicting appends — e.g. MergeOnEncrypted/compaction or delete commit validating no new files added to affected partitions since its starting snapshot.

Common situations: Concurrent Spark/Flink jobs writing to the same partition; DELETE/UPDATE statements racing with batch appends; retrying a stale commit after another job already appended.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/c6c91d4bc82927e2. Report an issue: GitHub.