apache/iceberg · error · UncheckedIOException

Failed to validate no deleted data files matching %s

Error message

Failed to validate no deleted data files matching %s

What it means

UncheckedIOException wrapping an IOException that occurred while scanning manifests to verify no concurrently deleted data files match the given filter. The delete-conflict validation could not complete because manifest data could not be read, so the commit is aborted. The real cause is available on the chained exception.

Source

Thrown at core/src/main/java/org/apache/iceberg/MergingSnapshotProducer.java:695

   * @param dataFilter an expression used to find deleted data files
   * @param parent ending snapshot on the branch being validated
   */
  protected void validateDeletedDataFiles(
      TableMetadata base, Long startingSnapshotId, Expression dataFilter, Snapshot parent) {
    CloseableIterable<ManifestEntry<DataFile>> conflictEntries =
        deletedDataFiles(base, startingSnapshotId, dataFilter, null, parent);

    try (CloseableIterator<ManifestEntry<DataFile>> conflicts = conflictEntries.iterator()) {
      if (conflicts.hasNext()) {
        throw new ValidationException(
            "Found conflicting deleted files that can contain records matching %s: %s",
            dataFilter,
            Iterators.toString(
                Iterators.transform(conflicts, entry -> entry.file().location().toString())));
      }

    } catch (IOException e) {
      throw new UncheckedIOException(
          String.format("Failed to validate no deleted data files matching %s", dataFilter), e);
    }
  }

  /**
   * Validates that no files matching a filter have been deleted from the table since a starting
   * snapshot.
   *
   * @param base table metadata to validate
   * @param startingSnapshotId id of the snapshot current at the start of the operation
   * @param partitionSet a partition set used to find deleted data files
   * @param parent ending snapshot on the branch being validated
   */
  protected void validateDeletedDataFiles(
      TableMetadata base, Long startingSnapshotId, PartitionSet partitionSet, Snapshot parent) {
    CloseableIterable<ManifestEntry<DataFile>> conflictEntries =
        deletedDataFiles(base, startingSnapshotId, null, partitionSet, parent);

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. Retry the commit with a retry policy (Tasks.foreach with exponential backoff) once storage is healthy.
2. Diagnose and fix the underlying IOException cause (connectivity, credentials, permissions) from the exception chain.
3. Check that manifest files for the validation start snapshot still exist and are readable.
4. Shorten the time between refresh and commit to shrink the manifest-scan window.

Example fix

// before
table.newOverwrite().validateNoDeletedFiles(oldSnapshotId).commit(); // UncheckedIOException on transient S3 error
// after
Tasks.foreach(() -> {
      table.refresh();
      table.newOverwrite().validateNoDeletedFiles(table.currentSnapshot().snapshotId()).commit();
    })
    .retry(3).exponentialBackoff(200, 10000, 60000)
    .run();

// pre-check manifest readability before commit
manifestsTable.entries(); // touch validation inputs via a metadata table read

try {
  operation.commit();
} catch (UncheckedIOException e) {
  // fix underlying IOException cause, then retry
}

When it happens

Trigger: Calling validateNoDeletedFiles (filter variant) during a commit where iterating deletedDataFiles manifest entries throws IOException from FileIO (network/storage failure).

Common situations: S3 503 slowdowns or credential expiry during commit validation; HDFS/DataNode failures; very long validation windows increasing exposure to transient storage faults.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

• Failed to validate required files exist
• Failed to validate no appends matching %s
• Found conflicting deleted files that can contain records mat
• Found conflicting deleted files that can apply to records ma
• Cannot commit, missing data files: %s

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/21ddb40f8f01e911. Report an issue: GitHub.