{"record":{"id":"6dfa1b3b12adc823","repo":"apache/seatunnel","slug":"close-file-output-stream-failed-6dfa1b","errorCode":null,"errorMessage":"Close file output stream {} failed","messagePattern":"Close file output stream (.+?) failed","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/sink/writer/JsonWriteStrategy.java","lineNumber":107,"sourceCode":"        }\n    }\n\n    @Override\n    public void finishAndCloseFile() {\n        beingWrittenOutputStream.forEach(\n                (key, value) -> {\n                    try {\n                        value.flush();\n                    } catch (IOException e) {\n                        throw new FileConnectorException(\n                                CommonErrorCodeDeprecated.FLUSH_DATA_FAILED,\n                                String.format(\"Flush data to this file [%s] failed\", key),\n                                e);\n                    } finally {\n                        try {\n                            value.close();\n                        } catch (IOException e) {\n                            log.warn(\"Close file output stream {} failed\", key, e);\n                        }\n                    }\n                    needMoveFiles.put(key, getTargetLocation(key));\n                });\n        beingWrittenOutputStream.clear();\n        isFirstWrite.clear();\n    }\n\n    @Override\n    public FSDataOutputStream getOrCreateOutputStream(@NonNull String filePath) {\n        FSDataOutputStream fsDataOutputStream = beingWrittenOutputStream.get(filePath);\n        if (fsDataOutputStream == null) {\n            try {\n                switch (compressFormat) {\n                    case LZO:\n                        LzopCodec lzo = new LzopCodec();\n                        OutputStream out =\n                                lzo.createOutputStream(","sourceCodeStart":89,"sourceCodeEnd":125,"githubUrl":"https://github.com/apache/seatunnel/blob/cf67b549a7a6c35fa0beb12d83c62892427ea919/seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/sink/writer/JsonWriteStrategy.java#L89-L125","documentation":"Logged in JsonWriteStrategy.finishAndCloseFile when closing an output stream for a completed JSON file throws an IOException. The warning means stream cleanup failed (broken FS connection, deleted file, storage error); the file is still queued for move to its target location. Job failure is not immediate but data in that file may need verification.","triggerScenarios":"At commit/close time, value.close() on an FSDataOutputStream in beingWrittenOutputStream throws IOException — typically due to HDFS connection loss, S3/OSS transient errors, or the target file being removed concurrently.","commonSituations":"Long-running jobs with expired Hadoop tokens; distributed FS rate limiting or 5xx during finalize; concurrent cleanup of the staging directory; disk-full conditions surfacing at close.","solutions":["Read the logged stack trace to find the root IOException and address the storage issue","Rerun the job; checkpoint recovery will regenerate the affected files","Ensure the staging path is not cleaned by external jobs and permissions are correct","Check cluster/storage health and tune FS client retry/timeout settings"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// preflight: ensure the output directory is writable and reachable\nFileSystem fs = outputPath.getFileSystem(conf);\nif (!fs.mkdirs(outputPath)) throw new IOException(\"cannot create \" + outputPath);\nfs.access(outputPath, FsAction.WRITE);","typeGuard":null,"tryCatchPattern":"try {\n    jobExecution.waitUntilFinish();\n} catch (Exception e) {\n    log.warn(\"Job ended with file close/commit issues, retrying\", e);\n    // resubmit; checkpoint recovery rewrites incomplete files\n}","preventionTips":["Enable FS client retries (fs.hdfs.impl retry settings / S3 retry config)","Set job checkpoints so partial files are recoverable","Verify target file integrity (size, record count) post-job","Avoid long jobs that outlive Kerberos/cloud credential lifetimes"],"tags":["ioexception","hadoop","file-close","json"],"backgroundTag":"file-write-failed","analyzedSha":"cf67b549a7a6c35fa0beb12d83c62892427ea919","analyzedAt":"2026-09-10T21:44:55.265Z","contentChangedAt":"2026-09-10T21:44:55.265Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}