Failed to determine live files in manifest {}. Retrying

Error message

Failed to determine live files in manifest {}. Retrying

What it means

ReachableFileCleanup.findFilesToDelete reads live file paths from each to-be-deleted manifest using ManifestFiles.readPaths inside Tasks with 3 retries and suppressed failure; the onFailure callback logs this warning per manifest. A manifest that cannot be read means some of its live data files are not identified, and cleanup will conservatively leave related files alone (or, per implementation, proceed with what it did collect).

Source

Thrown at core/src/main/java/org/apache/iceberg/ReachableFileCleanup.java:182

              }
            });

    return manifestFiles;
  }

  private Set<String> findFilesToDelete(
      Set<ManifestFile> manifestFilesToDelete,
      Set<ManifestFile> currentManifestFiles,
      Map<Integer, PartitionSpec> specsById) {
    Set<String> filesToDelete = ConcurrentHashMap.newKeySet();

    Tasks.foreach(manifestFilesToDelete)
        .retry(3)
        .suppressFailureWhenFinished()
        .executeWith(planExecutorService)
        .onFailure(
            (item, exc) ->
                LOG.warn(
                    "Failed to determine live files in manifest {}. Retrying", item.path(), exc))
        .run(
            manifest -> {
              try (CloseableIterable<String> paths =
                  ManifestFiles.readPaths(manifest, fileIO, specsById)) {
                paths.forEach(filesToDelete::add);
              } catch (IOException e) {
                throw new RuntimeIOException(e, "Failed to read manifest file: %s", manifest);
              }
            });

    if (filesToDelete.isEmpty()) {
      return filesToDelete;
    }

    try {
      Tasks.foreach(currentManifestFiles)
          .retry(3)

View on GitHub (pinned to 86d9c8fc54)

Solutions

Inspect the wrapped exception; restore or repair the unreadable manifest file if it should still exist.
Re-run expiration after fixing transient storage issues; the retry loop already attempted 3 times.
Verify no external lifecycle policy deletes Iceberg metadata files.
If manifests are unreadable leftovers from a broken copy, run RewriteTablePathUtil or recreate the table metadata.

Defensive patterns

Strategy: retry

Validate before calling

specsById.values().forEach(spec -> { /* ensure manifests reference valid spec ids */ });
Preconditions.checkArgument(io.newInputFile(manifest.path()).exists(), manifest.path());

Try / catch

try { expireSnapshots(); } catch (Exception e) { /* re-run after repairing manifest storage access */ }

Prevention

Keep compaction jobs from being killed mid-write to avoid corrupt manifests
Retry expiration with backoff on throttled object stores
Verify manifest files exist before expiration

When it happens

Trigger: Running expireSnapshots (dataFilesToDelete -> findFilesToDelete) when a manifest file targeted for deletion cannot be opened/parsed — missing file, corrupt Avro, throttled object store, wrong schema/spec mapping.

Common situations: Corrupt or truncated manifests from interrupted writes, external deletion of metadata by lifecycle rules, storage throttling during parallel reads, and spec ID mismatches after schema/partition evolution on copied tables.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/b5bc84f80c4a925e. Report an issue: GitHub.