apache/seatunnel · warning

Close file output stream

Error message

Close file output stream {} failed

What it means

Logged in JsonWriteStrategy.finishAndCloseFile when closing an output stream for a completed JSON file throws an IOException. The warning means stream cleanup failed (broken FS connection, deleted file, storage error); the file is still queued for move to its target location. Job failure is not immediate but data in that file may need verification.

Solutions

1. Read the logged stack trace to find the root IOException and address the storage issue
2. Rerun the job; checkpoint recovery will regenerate the affected files
3. Ensure the staging path is not cleaned by external jobs and permissions are correct
4. Check cluster/storage health and tune FS client retry/timeout settings

// preflight: ensure the output directory is writable and reachable
FileSystem fs = outputPath.getFileSystem(conf);
if (!fs.mkdirs(outputPath)) throw new IOException("cannot create " + outputPath);
fs.access(outputPath, FsAction.WRITE);

try {
    jobExecution.waitUntilFinish();
} catch (Exception e) {
    log.warn("Job ended with file close/commit issues, retrying", e);
    // resubmit; checkpoint recovery rewrites incomplete files
}

When it happens

Trigger: At commit/close time, value.close() on an FSDataOutputStream in beingWrittenOutputStream throws IOException — typically due to HDFS connection loss, S3/OSS transient errors, or the target file being removed concurrently.

Common situations: Long-running jobs with expired Hadoop tokens; distributed FS rate limiting or 5xx during finalize; concurrent cleanup of the staging directory; disk-full conditions surfacing at close.

Understand the failure class

Background: "failed to write file", "Could not save figure", "Error saving remote file" — file write failed: causes and fixes across languages and libraries — this error's family across 38 libraries.

Related errors

• Close file output stream
• Close file output stream
• Json file does not support this compress type
• AmazonDocumentDB option '' must be a valid BSON/JSON…
• CanalJson file does not support this compress type

AI-assisted analysis of apache/seatunnel@cf67b549a7 (2026-09-10). Data as JSON: /api/errors/6dfa1b3b12adc823. Report an issue: GitHub.

Appendix: source

Thrown at seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/sink/writer/JsonWriteStrategy.java:107

        }
    }

    @Override
    public void finishAndCloseFile() {
        beingWrittenOutputStream.forEach(
                (key, value) -> {
                    try {
                        value.flush();
                    } catch (IOException e) {
                        throw new FileConnectorException(
                                CommonErrorCodeDeprecated.FLUSH_DATA_FAILED,
                                String.format("Flush data to this file [%s] failed", key),
                                e);
                    } finally {
                        try {
                            value.close();
                        } catch (IOException e) {
                            log.warn("Close file output stream {} failed", key, e);
                        }
                    }
                    needMoveFiles.put(key, getTargetLocation(key));
                });
        beingWrittenOutputStream.clear();
        isFirstWrite.clear();
    }

    @Override
    public FSDataOutputStream getOrCreateOutputStream(@NonNull String filePath) {
        FSDataOutputStream fsDataOutputStream = beingWrittenOutputStream.get(filePath);
        if (fsDataOutputStream == null) {
            try {
                switch (compressFormat) {
                    case LZO:
                        LzopCodec lzo = new LzopCodec();
                        OutputStream out =
                                lzo.createOutputStream(

View on GitHub (pinned to cf67b549a7)