delete transaction directory

Error message

delete transaction directory [{}] failed after successful commit, ignore this cleanup error.

What it means

FileSinkAggregatedCommitter.commit, after successfully committing (renaming) files, deletes the temporary transaction directory as a best-effort cleanup. If hadoopFileSystemProxy.deleteFile throws, the error is logged with this message and deliberately NOT propagated, because the data files are already committed and cleanup failure must not fail the checkpoint.

Solutions

Safe to ignore for data correctness — files are already committed; just remove leftover tmp dirs manually
Check filesystem permissions on the staging/transaction parent directories
Ensure only one committer instance handles a given transaction (avoid duplicate commits from retries)
Configure lifecycle/TTL cleanup on the staging directory if leftovers accumulate

Defensive patterns

Strategy: try-catch

Validate before calling

// Before commit, verify staging dir state
if (!hadoopFileSystemProxy.fileExist(transactionDir)) { return; } // nothing to clean up

Try / catch

try { committer.commit(committables); } catch (Exception e) { /* real commit failures surface here; cleanup warnings never do — check logs for 'ignore this cleanup error' */ throw e; }

Prevention

Ensure staging/transaction parent dirs are writable by the job user
Avoid concurrent committers on the same transaction dir
On object stores, expect eventual-consistency cleanup warnings and schedule TTL cleanup

When it happens

Trigger: deleteFile on the transaction directory throws (permissions on the parent dir, directory already removed by another committer/retry, HDFS/S3 transient error) after a successful rename-based commit.

Common situations: Object stores (S3/OSS) with eventual consistency where a just-renamed path briefly appears still present; concurrent committer retries racing on the same transaction dir; HDFS permission changes on staging directories.

Understand the failure class

Background: "failed to write file", "Could not save figure", "Error saving remote file" — file write failed: causes and fixes across languages and libraries — this error's family across 38 libraries.

Related errors

AI-assisted analysis of apache/seatunnel@cf67b549a7 (2026-09-10). Data as JSON: /api/errors/5f5940e8b9a09388. Report an issue: GitHub.

Appendix: source

Thrown at seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/sink/commit/FileSinkAggregatedCommitter.java:75

        aggregatedCommitInfos.forEach(
                aggregatedCommitInfo -> {
                    try {
                        for (Map.Entry<String, LinkedHashMap<String, String>> entry :
                                aggregatedCommitInfo.getTransactionMap().entrySet()) {
                            for (Map.Entry<String, String> mvFileEntry :
                                    entry.getValue().entrySet()) {
                                // first rename temp file
                                hadoopFileSystemProxy.renameFile(
                                        mvFileEntry.getKey(), mvFileEntry.getValue(), true);
                            }
                            String transactionDir = entry.getKey();
                            // Data files are already committed after rename; tmp cleanup is
                            // best-effort and should not fail the whole checkpoint.
                            try {
                                hadoopFileSystemProxy.deleteFile(transactionDir);
                                registerTransactionParentDirectories(transactionDir);
                            } catch (Exception cleanupException) {
                                log.warn(
                                        "delete transaction directory [{}] failed after successful commit, ignore this cleanup error.",
                                        transactionDir,
                                        cleanupException);
                            }
                        }
                    } catch (Throwable e) {
                        log.error(
                                "commit aggregatedCommitInfo error, aggregatedCommitInfo = {} ",
                                aggregatedCommitInfo,
                                e);
                        errorAggregatedCommitInfoList.add(aggregatedCommitInfo);
                    }
                });
        return errorAggregatedCommitInfoList;
    }

    /**
     * The logic about how to combine commit message.

View on GitHub (pinned to cf67b549a7)