What causes "Failed to clean empty transaction parent directory: {}" in apache/seatunnel?

Check write permissions on the output path for the job's commit user

Failed to clean empty transaction parent directory

Error message

Failed to clean empty transaction parent directory: {}

What it means

A WARN from FileSinkAggregatedCommitter.cleanupEmptyDirectory: deleting a now-empty per-transaction parent directory (created for staging transaction files) failed with an IOException after commit. Committed data files are safe; only an empty directory may linger in the output path.

Solutions

Check write permissions on the output path for the job's commit user
List the directory — if non-empty, clean orphaned files from failed transactions manually
Retry after transient storage errors; safe to delete manually once all files are committed
On S3, verify bucket throttling (429s) is not interrupting cleanup

Defensive patterns

Strategy: try-catch

Validate before calling

// pre-commit check: ensure commit user can write the output tree
// hadoop fs -test -w /output/parent && echo writable

Try / catch

// safe to ignore for committed data; cleanup pattern:
// try { fs.deleteEmptyDirectory(dir); } catch (IOException e) { scheduleManualCleanup(dir); }

Prevention

Grant the sink user recursive write/delete on the output path
Clean orphaned transaction dirs from crashed runs before re-running
Avoid heavy S3 parallelism that triggers throttled deletes during commit

When it happens

Trigger: cleanupTransactionParentDirectories -> cleanupEmptyDirectory calls hadoopFileSystemProxy.deleteEmptyDirectory(dir); the filesystem throws IOException due to permission issues, transient NameNode/S3 errors, or the directory not actually being empty (orphaned files from a failed earlier transaction).

Common situations: S3 eventual consistency / throttling during commit; HDFS permission mismatch between the committer user and directory owner; a crashed prior job left files in the transaction directory so it is not empty.

Understand the failure class

Background: "failed to write file", "Could not save figure", "Error saving remote file" — file write failed: causes and fixes across languages and libraries — this error's family across 38 libraries.

Related errors

AI-assisted analysis of apache/seatunnel@cf67b549a7 (2026-09-10). Data as JSON: /api/errors/cc47372483c8f26c. Report an issue: GitHub.

Appendix: source

Thrown at seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/sink/commit/FileSinkAggregatedCommitter.java:209

        if (jobDir == null) {
            return;
        }
        cleanupEmptyDirectory(uuidDir.toString());
        cleanupEmptyDirectory(jobDir.toString());
    }

    private void cleanupPendingTransactionParentDirectories() {
        pendingUuidDirectories.forEach(this::cleanupEmptyDirectory);
        pendingJobDirectories.forEach(this::cleanupEmptyDirectory);
        pendingUuidDirectories.clear();
        pendingJobDirectories.clear();
    }

    private void cleanupEmptyDirectory(String directory) {
        try {
            hadoopFileSystemProxy.deleteEmptyDirectory(directory);
        } catch (IOException e) {
            log.warn("Failed to clean empty transaction parent directory: {}", directory, e);
        }
    }
}

View on GitHub (pinned to cf67b549a7)