apache/seatunnel · warning

Failed to read manifest {}, assuming files not yet committed

Error message

Failed to read manifest {}, assuming files not yet committed

What it means

This warning is logged during Iceberg commit deduplication (areFilesAlreadyCommitted) when a manifest file cannot be read due to an IOException. The committer treats unreadable manifests as evidence that the files were not yet committed (conservative fallback) so recovery/commit proceeds instead of failing. This typically happens when a manifest listed in a snapshot was deleted by expiry or is not yet visible (eventual consistency on object stores).

Source

Thrown at seatunnel-connectors-v2/connector-iceberg/src/main/java/org/apache/seatunnel/connectors/seatunnel/iceberg/sink/commit/IcebergFilesCommitter.java:150

                            ManifestFiles.read(manifest, table.io())) {
                        for (DataFile file : reader) {
                            if (pendingPaths.contains(file.path().toString())) {
                                return true;
                            }
                        }
                    }
                } else {
                    try (CloseableIterable<DeleteFile> reader =
                            ManifestFiles.readDeleteManifest(manifest, table.io(), table.specs())) {
                        for (DeleteFile file : reader) {
                            if (pendingPaths.contains(file.path().toString())) {
                                return true;
                            }
                        }
                    }
                }
            } catch (IOException e) {
                log.warn(
                        "Failed to read manifest {}, assuming files not yet committed",
                        manifest.path(),
                        e);
            }
        }
        return false;
    }

    private void commit(
            TableIdentifier tableIdentifier, List<WriteResult> results, long checkpointId) {
        List<DataFile> dataFiles =
                results.stream()
                        .filter(payload -> payload.getDataFiles() != null)
                        .flatMap(payload -> payload.getDataFiles().stream())
                        .filter(dataFile -> dataFile.recordCount() > 0)
                        .collect(toList());

        List<DeleteFile> deleteFiles =

View on GitHub (pinned to cf67b549a7)

Solutions

1. Verify the Iceberg table's snapshot expiry/retention settings and ensure they are longer than the job's checkpoint interval
2. Check filesystem/object-store permissions and network stability for the warehouse path
3. Retry the job — the warning is non-fatal by design and the commit proceeds
4. If it recurs constantly, inspect table metadata for orphaned manifest references (e.g. after a failed expireSnapshots)

Example fix

// before
// table expires snapshots aggressively
expireSnapshots.olderThan(60_000).execute();
// after
// retain snapshots longer than job checkpoint interval
expireSnapshots.olderThan(System.currentTimeMillis() - 86400_000).execute();

// before committing, check manifest readability
ManifestFile mf = ...;
try (FileIO io = table.io()) {
    io.newInputFile(mf.path());
} catch (IOException e) { /* treat as not committed / retry */ }

try { areFilesAlreadyCommitted(...); } catch (IOException e) {
    log.warn("Manifest unreadable, will retry commit", e);
    retryWithBackoff(() -> areFilesAlreadyCommitted(...));
}

When it happens

Trigger: During isAlreadyCommitted after restore/restart, the committer iterates snapshot manifests and IOException occurs reading a manifest (manifest expired/removed by table maintenance, transient S3/HDFS read failure, or permissions issue).

Common situations: Job restored from checkpoint after snapshot expiry removed old manifests; object-store eventual consistency causing a just-written manifest to be temporarily invisible; HDFS NameNode failover or transient network errors.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

• Failed to close equality delta writer
• FILE_SCAN_SPLIT_FAILED
• Error adding Hadoop resource {}, resource was not added
• Circular condition chain detected: '%s' already exists in th
• Condition for option '%s' has a null operator

AI-assisted analysis of apache/seatunnel@cf67b549a7 (2026-09-10). Data as JSON: /api/errors/fe9904272ff27391. Report an issue: GitHub.