Failed to get added files: this may cause orphaned data file

Error message

Failed to get added files: this may cause orphaned data files

What it means

In findFilesToDelete, manifests that must be reverted (files added after the 'before' state) are scanned so their ADDED files can be deleted. If such a scan fails after retries, this WARN 'Failed to get added files: this may cause orphaned data files' is logged and the run continues. Files added by reverted snapshots may then never be deleted — orphaned data files.

Source

Thrown at core/src/main/java/org/apache/iceberg/IncrementalFileCleanup.java:319

                  // deleted
                  if (entry.status() == ManifestEntry.Status.DELETED
                      && !validIds.contains(entry.snapshotId())) {
                    // use toString to ensure the path will not change (Utf8 is reused)
                    filesToDelete.add(entry.file().location());
                  }
                }
              } catch (IOException e) {
                throw new RuntimeIOException(e, "Failed to read manifest file: %s", manifest);
              }
            });

    Tasks.foreach(manifestsToRevert)
        .retry(3)
        .suppressFailureWhenFinished()
        .executeWith(planExecutorService)
        .onFailure(
            (item, exc) ->
                LOG.warn("Failed to get added files: this may cause orphaned data files", exc))
        .run(
            manifest -> {
              // the manifest has deletes, scan it to find files to delete
              try (ManifestReader<?> reader = ManifestFiles.open(manifest, fileIO, specsById)) {
                for (ManifestEntry<?> entry : reader.entries()) {
                  // delete any ADDED file from manifests that were reverted
                  if (entry.status() == ManifestEntry.Status.ADDED) {
                    // use toString to ensure the path will not change (Utf8 is reused)
                    filesToDelete.add(entry.file().location());
                  }
                }
              } catch (IOException e) {
                throw new RuntimeIOException(e, "Failed to read manifest file: %s", manifest);
              }
            });

    return filesToDelete;
  }

View on GitHub (pinned to 86d9c8fc54)

Solutions

Inspect the logged exception for the manifest path and fix access/corruption issues.
Rerun expiration (cleanup is idempotent and incremental) after access issues are resolved.
Avoid concurrent table expiration/maintenance to prevent manifest races.
Use full cleanup or remove-orphan-files to reclaim files skipped by this run.

Defensive patterns

Strategy: retry

Validate before calling

// ensure all manifests to be reverted exist and are openable
manifestsToRevert.forEach(m -> Preconditions.checkArgument(io.newInputFile(m.path()).exists()));

Try / catch

try {
  table.expireSnapshots().execute();
} catch (RuntimeException e) {
  LOG.warn("Revert-scan failures may orphan added files; run orphan cleanup", e);
}

Prevention

Run a single expiration job per table at a time.
Use consistent storage to prevent stale manifest listings.
After failures, run remove-orphan-files with an appropriate olderThan window.
Fix underlying IO errors surfaced in the WARN stack trace before rerunning.

When it happens

Trigger: filesToDelete -> findFilesToDelete processes manifestsToRevert; ManifestFiles.open or entry iteration throws for a manifest (IO error, missing object, corruption) so the ADDED files cannot be enumerated.

Common situations: Overlapping/concurrent expiration deleting manifests mid-run; eventual consistency showing stale listings; object-store errors under parallel planExecutorService load.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/0105b2c91f68bb4f. Report an issue: GitHub.