What causes "Failed to validate required files exist" in apache/iceberg?

Retry the commit with a retry policy once the storage backend is reachable.

apache/iceberg · error · UncheckedIOException

Failed to validate required files exist

Error message

Failed to validate required files exist

What it means

UncheckedIOException wrapping an IOException that occurred while checking that data files required by delete files still exist at commit time. The existence validation could not finish because manifest scanning failed on I/O, so the commit aborts. The underlying cause is in the chained exception.

Source

Thrown at core/src/main/java/org/apache/iceberg/MergingSnapshotProducer.java:864

                        && requiredDataFiles.contains(entry.file().location()))
            .specsById(base.specsById())
            .ignoreExisting();

    if (conflictDetectionFilter != null) {
      matchingDeletesGroup.filterData(conflictDetectionFilter);
    }

    try (CloseableIterator<ManifestEntry<DataFile>> deletes =
        matchingDeletesGroup.entries().iterator()) {
      if (deletes.hasNext()) {
        throw new ValidationException(
            "Cannot commit, missing data files: %s",
            Iterators.toString(
                Iterators.transform(deletes, entry -> entry.file().location().toString())));
      }

    } catch (IOException e) {
      throw new UncheckedIOException("Failed to validate required files exist", e);
    }
  }

  // validates there are no concurrently added DVs for referenced data files
  protected void validateAddedDVs(
      TableMetadata base,
      Long startingSnapshotId,
      Expression conflictDetectionFilter,
      Snapshot parent) {
    // skip if there is no current table state or this operation doesn't add new DVs
    if (parent == null || dvsByReferencedFile.isEmpty()) {
      return;
    }

    Pair<List<ManifestFile>, Set<Long>> history =
        validationHistory(
            base,
            startingSnapshotId,

View on GitHub (pinned to 86d9c8fc54)

Solutions

Retry the commit with a retry policy once the storage backend is reachable.
Fix the root IOException cause (network, credentials, missing files) from the exception chain.
Ensure snapshot expiration jobs are not deleting files concurrently with commits.
Refresh the table right before commit to keep the validation window short.

Example fix

// before
rowDelta.commit(); // UncheckedIOException: Failed to validate required files exist
// after
Tasks.foreach(rowDelta::commit)
    .retry(3).exponentialBackoff(100, 10000, 60000)
    .throwFailureWhenFinished()
    .run();

Defensive patterns

Strategy: retry

Validate before calling

// pre-check required data file existence via FileIO
for (String path : referencedPaths) { io.newInputFile(path).exists(); }

Try / catch

try {
  rowDelta.commit();
} catch (UncheckedIOException e) {
  // inspect e.getCause(); retry once storage is healthy
}

Prevention

Retry commits with backoff around storage faults
Keep expiration jobs isolated from conflicting commits
Refresh close to commit to shrink the validation window

When it happens

Trigger: Committing an operation that calls validateRequiredFilesExist (e.g., RowDelta with delete files) where iterating the manifest entries throws IOException from the underlying FileIO.

Common situations: Storage outages (S3/HDFS) during commit; expired credentials; data/manifest files deleted by concurrent snapshot expiration during validation.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/6a3fef9e23137126. Report an issue: GitHub.