apache/iceberg · error · ValidationException

Cannot delete data files %s that are referenced by new delet

Error message

Cannot delete data files %s that are referenced by new delete files

What it means

RowDelta operations both append delete (position/deletion-vector) files and can delete data files; this validation rejects commits where a data file being deleted is referenced by a delete file in the same operation. Deleting the data file would orphan the new delete file's positions, corrupting row-level deletion semantics.

Source

Thrown at core/src/main/java/org/apache/iceberg/BaseRowDelta.java:189

      validateAddedDVs(base, startingSnapshotId, conflictDetectionFilter, parent);
    }
  }

  /**
   * Validates that the data files removed in this commit do not overlap with data files with delete
   * files added
   */
  @SuppressWarnings("CollectionUndefinedEquality")
  private void validateNoConflictingFileAndPositionDeletes() {
    List<CharSequence> deletedFileWithNewDVs =
        removedDataFiles.stream()
            .map(DataFile::path)
            .filter(referencedDataFiles::contains)
            .collect(Collectors.toList());

    if (!deletedFileWithNewDVs.isEmpty()) {
      throw new ValidationException(
          "Cannot delete data files %s that are referenced by new delete files",
          deletedFileWithNewDVs);
    }
  }
}

View on GitHub (pinned to 86d9c8fc54)

Solutions

  1. Split the work: commit the RowDelta that adds delete files first, then delete the data files in a separate, later operation (e.g. via deleteFiles/rewriteDataFiles)
  2. Refresh the table and re-validate that targeted data file paths are not in newly added delete files before deleting
  3. Use rewriteDataFiles instead of manual RowDelta so compaction and deletes are coordinated
  4. Coordinate merge-on-read writers and compaction jobs to avoid racing on the same files

Example fix

// before: same RowDelta does both
Transaction tx = table.newTransaction();
table.newRowDelta().addDeletes(deleteFile).deleteDataFile(dataFile).commit();
// after: sequence operations
table.newRowDelta().addDeletes(deleteFile).commit(); // snapshot A
table.refresh();
table.newDelete().deleteFile(dataFile).commit(); // snapshot B
Defensive patterns

Strategy: validation

Validate before calling

Set<CharSequence> referenced = newDeleteFiles.stream().flatMap(df -> df.referencedDataFiles().stream()).collect(Collectors.toSet());
boolean conflict = dataFilesToDelete.stream().anyMatch(f -> referenced.contains(f.path()));
if (conflict) throw new IllegalStateException("data files referenced by new delete files");

Try / catch

try { rowDelta.commit(); } catch (ValidationException e) { table.refresh(); reapplyAsSeparateCommits(); }

Prevention

When it happens

Trigger: Calling newRowDelta() that both (a) addsDeleteFile / DV writers referencing data files and (b) deletes data files with deleteRows/deleteDataFiles for those same paths in one commit.

Common situations: Compaction jobs that remove rewritten data files while a concurrent merge-on-read writer publishes delete files targeting them; misconfigured maintenance pipelines issuing both changes in one RowDelta instead of separate snapshots.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/920c5ab0c88239e5. Report an issue: GitHub.