apache/beam · error · RuntimeException

Failed to collect file statuses for snapshot

Error message

Failed to collect file statuses for snapshot {}

What it means

buildFileStatusBySnapshot iterates a snapshot's manifest entries to map file locations to entry statuses for changelog snapshots. Any exception while reading manifest content (IO error, corrupt manifest, deserialization failure) is rethrown as a RuntimeException naming the snapshot id.

Solutions

1. Check the wrapped cause for the underlying IO/deserialization error and verify the manifest file exists and is readable on the filesystem
2. Retry the scan; transient object-store failures (503, throttling) commonly cause this
3. Verify no concurrent expireSnapshots/rewriteJobs deleted the manifests referenced by the scanned snapshots; pause retention cleanup or rerun against a valid snapshot range

Example fix

// before
scan = table.newIncrementalChangelogScan().fromSnapshotId(oldSnap).toSnapshotId(headSnap); // oldSnap manifests expired
// after
long validFrom = table.currentSnapshot().getParentSnapshotId();
scan = table.newIncrementalChangelogScan().fromSnapshotId(validFrom).toSnapshotId(table.currentSnapshot().snapshotId());

// before scan: confirm all snapshots in range still exist
for (Snapshot s : snapshotsInRange) {
  checkState(table.snapshot(s.snapshotId()) != null, "snapshot expired: " + s.snapshotId());
}

try {
  table.newIncrementalChangelogScan()...planFiles();
} catch (RuntimeException e) {
  if (e.getMessage().startsWith("Failed to collect file statuses")) {
    // retry with backoff; if persistent, widen retention and re-run
  } else throw e;
}

When it happens

Trigger: Reading manifest entries of snapshot snapshotId() during an incremental changelog scan when the manifest file is unreadable, deleted, or its data/sequence metadata cannot be deserialized.

Common situations: Underlying files removed by expireSnapshots/rewriteManifests while the scan runs; GCS/S3/HDFS transient I/O failures; corrupt manifests from a failed commit.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

• Failed to plan deleted rows tasks
• Failed to read delete manifest
• A schema is required to write non-schema'd data.
• A sink must inherit iobase.Sink, iobase.NativeSink, or be a…
• Adding required columns is not yet supported. Encountered…

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/2f439fca9a2dfcdb. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/io/iceberg/src/main/java/org/apache/iceberg/BeamBaseIncrementalChangelogScan.java:443

              if (!changedDataManifests.isEmpty()) {
                ManifestGroup changedGroup =
                    new ManifestGroup(table().io(), changedDataManifests, ImmutableList.of())
                        .specsById(table().specs())
                        .caseSensitive(isCaseSensitive())
                        .select(scanColumns())
                        .filterData(filter())
                        .ignoreExisting()
                        .columnsToKeepStats(columnsToKeepStats());

                try (CloseableIterable<ManifestEntry<DataFile>> entries = changedGroup.entries()) {
                  for (ManifestEntry<DataFile> entry : entries) {
                    if (changelogSnapshotIds.contains(entry.snapshotId())) {
                      fileStatuses.put(entry.file().location(), entry.status());
                      localAffected.add(entry.file().specId(), entry.file().partition());
                    }
                  }
                } catch (Exception e) {
                  throw new RuntimeException(
                      "Failed to collect file statuses for snapshot " + snapshot.snapshotId(), e);
                }
              }

              fileStatusBySnapshot.put(snapshot.snapshotId(), fileStatuses);
              localPartitionsQueue.add(localAffected);
            });

    PartitionSet globalAffected = PartitionSet.create(table().specs());
    for (PartitionSet local : localPartitionsQueue) {
      globalAffected.addAll(local);
    }

    return Pair.of(fileStatusBySnapshot, globalAffected);
  }

  private List<ManifestFile> pruneManifestsByAffectedPartitions(
      List<ManifestFile> manifests, PartitionSet affectedPartitions) {

View on GitHub (pinned to 12126d8942)