apache/iceberg · error · ValidationException

Cannot commit file that conflicts with existing partition: %

Error message

Cannot commit file that conflicts with existing partition: %s

What it means

ReplacePartitions overwrites entire partitions by deleting all rows of the incoming files' partitions. When super.apply() detects (via ManifestFilterManager.DeleteException) that the commit would delete a partition that contains files not covered by the replace — e.g. a concurrent commit added files to that partition — it throws this ValidationException to preserve atomicity.

Source

Thrown at core/src/main/java/org/apache/iceberg/BaseReplacePartitions.java:122

            currentMetadata, startingSnapshotId, Expressions.alwaysTrue(), parent);
      } else {
        validateDeletedDataFiles(currentMetadata, startingSnapshotId, replacedPartitions, parent);
        validateNoNewDeleteFiles(currentMetadata, startingSnapshotId, replacedPartitions, parent);
      }
    }
  }

  @Override
  public List<ManifestFile> apply(TableMetadata base, Snapshot snapshot) {
    if (dataSpec().isUnpartitioned()) {
      // replace all data in an unpartitioned table
      deleteByRowFilter(Expressions.alwaysTrue());
    }

    try {
      return super.apply(base, snapshot);
    } catch (ManifestFilterManager.DeleteException e) {
      throw new ValidationException(
          "Cannot commit file that conflicts with existing partition: %s", e.partition());
    }
  }
}

View on GitHub (pinned to 86d9c8fc54)

Solutions

Re-run the job: re-plan the ReplacePartitions against the latest snapshot so the conflicting files are covered by the delete filter.
Use a retry loop on ValidationException/CommitFailedException around the overwrite.
Switch to overwrite(newOverwrite()) with explicit file-level overwrites when only some files should be replaced.
Ensure application-level locking so only one writer replaces a given partition at a time.

Example fix

// before
table.newReplacePartitions().appendFile(file).commit(); // throws on conflict
// after
Tasks.foreach(files)
    .retry(3)
    .exponentialBackoff(100, 3000)
    .run(f -> table.refresh().newReplacePartitions().appendFile(f).commit());

Defensive patterns

Strategy: retry

Validate before calling

// before committing, confirm the target partitions have no un-replaced files
List<DataFile> partitionFiles = Files.tableFiles(table)
    .filter(f -> targetSpec.partitionType().equals(f.partitionSpec(specId).partitionType()))
    .collect(Collectors.toList());
// ensure all partitionFiles are being overwritten by the ReplacePartitions

Try / catch

try {
  table.newReplacePartitions().appendFiles(files).commit();
} catch (ValidationException e) {
  if (e.getMessage().startsWith("Cannot commit file that conflicts with existing partition")) {
    table.refresh(); // re-plan against latest snapshot and retry
  } else throw e;
}

Prevention

Retry ReplacePartitions with a refreshed table on conflict
Serialize partition overwrites with application-level locks
Avoid overwriting partitions while other writers append to them
Prefer OverwriteFiles for file-level (not whole-partition) replacement

When it happens

Trigger: Committing a ReplacePartitions operation whose data files touch partitions that contain other data files not being replaced (typical with non-identity/dynamic partition layouts or a concurrent writer that added files to the same partition between plan and commit).

Common situations: Two jobs concurrently overwriting the same partition; overwriting a partition where files were written with a different spec; replace-partitions against tables where rows of one partition span multiple writers.

Understand the failure class

Background: Conflicting config options: "cannot be used together" — configuration validation errors across open-source libraries — this error's family across 162 libraries.

Related errors

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/5d8f9980f796451b. Report an issue: GitHub.