apache/seatunnel · warning

Close file output stream

Error message

Close file output stream {} failed

What it means

Logged in DebeziumJsonWriteStrategy.finishAndCloseFile when closing an output stream for a finished file throws an IOException. Data was already flushed (flush failure is reported separately); this warning indicates the stream close itself failed, e.g. because the underlying Hadoop FS connection is broken or the file was deleted/moved concurrently. The file is still registered in needMoveFiles for commit.

Solutions

1. Inspect the attached stack trace for the underlying IOException cause and fix the storage/network issue
2. Retry the job; SeaTunnel checkpoint/restart will rewrite the affected files
3. Verify the staging directory is not concurrently cleaned and the user has write permissions
4. Check storage backend health (DataNodes, S3 endpoint) and increase timeouts/retry settings

// check storage reachability and staging dir writability before the job
FileSystem fs = FileSystem.get(conf);
Path staging = new Path(stagingDir);
if (!fs.exists(staging) || !fs.getFileStatus(staging).getPermission().getAction().contains(org.apache.hadoop.fs.permission.FsAction.WRITE)) { throw new IllegalStateException("staging dir not writable: " + staging); }

// rely on framework retry; if wrapping, log-and-continue with verification
try {
    strategy.finishAndCloseFile(...);
} catch (Exception e) {
    log.warn("File commit issue, will retry job/verify files", e);
    // verify target files exist and sizes > 0 before declaring success
}

When it happens

Trigger: During sink close/commit, finishAndCloseFile iterates beingWrittenOutputStream and calls value.close(); the FSDataOutputStream.close throws IOException due to a failed HDFS/DataNode connection, deleted staging file, or quota/permission issue surfacing at close time.

Common situations: HDFS DataNode unavailability or network partitions at job end; distributed FS (S3/OSS) transient 5xx on finalize; file removed by a cleanup job before close; Kerberos token expiry on long jobs.

Understand the failure class

Background: "failed to write file", "Could not save figure", "Error saving remote file" — file write failed: causes and fixes across languages and libraries — this error's family across 38 libraries.

Related errors

• Close file output stream
• Close file output stream
• Failed to get file system
• CanalJson file does not support this compress type
• Cannot rename file from

AI-assisted analysis of apache/seatunnel@cf67b549a7 (2026-09-10). Data as JSON: /api/errors/e6386194fd4dd444. Report an issue: GitHub.

Appendix: source

Thrown at seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/sink/writer/DebeziumJsonWriteStrategy.java:115

        }
    }

    @Override
    public void finishAndCloseFile() {
        beingWrittenOutputStream.forEach(
                (key, value) -> {
                    try {
                        value.flush();
                    } catch (IOException e) {
                        throw new FileConnectorException(
                                CommonErrorCodeDeprecated.FLUSH_DATA_FAILED,
                                String.format("Flush data to this file [%s] failed", key),
                                e);
                    } finally {
                        try {
                            value.close();
                        } catch (IOException e) {
                            log.warn("Close file output stream {} failed", key, e);
                        }
                    }
                    needMoveFiles.put(key, getTargetLocation(key));
                });
        beingWrittenOutputStream.clear();
        isFirstWrite.clear();
    }

    @Override
    public FSDataOutputStream getOrCreateOutputStream(@NonNull String filePath) {
        FSDataOutputStream fsDataOutputStream = beingWrittenOutputStream.get(filePath);
        if (fsDataOutputStream == null) {
            try {
                switch (compressFormat) {
                    case LZO:
                        LzopCodec lzo = new LzopCodec();
                        OutputStream out =
                                lzo.createOutputStream(

View on GitHub (pinned to cf67b549a7)