What causes "Cannot commit, missing data files: %s" in apache/iceberg?

Refresh the table and restart the operation against the current snapshot so delete files reference existing data files.

apache/iceberg · error · ValidationException

Cannot commit, missing data files: %s

Error message

Cannot commit, missing data files: %s

What it means

ValidationException thrown during commit when data files referenced by delete files (equality or position deletes) no longer exist — i.e., some referenced data files are missing from the current table state. Iceberg aborts because applying the deletes would be against data files that have been removed or rewritten by a concurrent operation. It usually indicates conflicting concurrent writers or an operation based on a stale snapshot.

Source

Thrown at core/src/main/java/org/apache/iceberg/MergingSnapshotProducer.java:857

    ManifestGroup matchingDeletesGroup =
        new ManifestGroup(ops().io(), manifests, ImmutableList.of())
            .filterManifestEntries(
                entry ->
                    entry.status() != ManifestEntry.Status.ADDED
                        && newSnapshots.contains(entry.snapshotId())
                        && requiredDataFiles.contains(entry.file().location()))
            .specsById(base.specsById())
            .ignoreExisting();

    if (conflictDetectionFilter != null) {
      matchingDeletesGroup.filterData(conflictDetectionFilter);
    }

    try (CloseableIterator<ManifestEntry<DataFile>> deletes =
        matchingDeletesGroup.entries().iterator()) {
      if (deletes.hasNext()) {
        throw new ValidationException(
            "Cannot commit, missing data files: %s",
            Iterators.toString(
                Iterators.transform(deletes, entry -> entry.file().location().toString())));
      }

    } catch (IOException e) {
      throw new UncheckedIOException("Failed to validate required files exist", e);
    }
  }

  // validates there are no concurrently added DVs for referenced data files
  protected void validateAddedDVs(
      TableMetadata base,
      Long startingSnapshotId,
      Expression conflictDetectionFilter,
      Snapshot parent) {
    // skip if there is no current table state or this operation doesn't add new DVs
    if (parent == null || dvsByReferencedFile.isEmpty()) {

View on GitHub (pinned to 86d9c8fc54)

Solutions

Refresh the table and restart the operation against the current snapshot so delete files reference existing data files.
Re-run the MERGE/DELETE after the compaction job finishes; sequence rewrite and DML jobs.
Use validation (conflictDetectionFilter) correctly and retry the whole job rather than forcing the commit.
Avoid snapshot expiration of files still referenced by in-flight operations.

Example fix

// before
RowDelta rd = table.newRowDelta().addDeletes(deleteFile);
rd.conflictDetectionFilter(filter);
rd.validateNoConflictingDeletes(startSnapshotId);
rd.commit(); // Cannot commit, missing data files
// after
table.refresh();
RowDelta rd = table.newRowDelta().addDeletes(recomputedDeleteFile); // recompute deletes on current snapshot
table.currentSnapshot(); // ensure base data files exist in current state
rd.commit();

Defensive patterns

Strategy: retry

Validate before calling

table.refresh();
// verify referenced data files still exist before committing deletes
Snapshot current = table.currentSnapshot();

Try / catch

try {
  rowDelta.commit();
} catch (ValidationException e) {
  table.refresh();
  // recompute delete files against current data files and retry
}

Prevention

Recompute delete files on the current snapshot before commit
Don't run compaction/rewrite concurrently with MERGE/DELETE that reference the same files
Avoid expiring snapshots that in-flight operations depend on

When it happens

Trigger: Committing RowDelta/operation with conflictDetectionFilter set while concurrent commits deleted the data files this operation's delete files reference; detected via matchingDeletesGroup containing those missing files.

Common situations: Spark MERGE/DELETE running concurrently with compaction (rewriteDataFiles) that removed the referenced files; writer holding a stale snapshot for a long time; expired snapshots removing referenced files.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/0ed7556f368d4aa7. Report an issue: GitHub.