{"record":{"id":"f57c945bb0318205","repo":"apache/seatunnel","slug":"flush-data-failed-f57c94","errorCode":"FLUSH_DATA_FAILED","errorMessage":"Flush data to this file [%s] failed","messagePattern":"Flush data to this file \\[(.+?)\\] failed","errorType":"error_code","errorClass":"FileConnectorException","httpStatus":null,"severity":"error","filePath":"seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/sink/writer/CanalJsonWriteStrategy.java","lineNumber":107,"sourceCode":"            if (isFirstWrite.get(filePath)) {\n                isFirstWrite.put(filePath, false);\n            } else {\n                fsDataOutputStream.write(rowDelimiter);\n            }\n            fsDataOutputStream.write(rowBytes);\n        } catch (IOException e) {\n            throw CommonError.fileOperationFailed(\"CanalJsonFile\", \"write\", filePath, e);\n        }\n    }\n\n    @Override\n    public void finishAndCloseFile() {\n        beingWrittenOutputStream.forEach(\n                (key, value) -> {\n                    try {\n                        value.flush();\n                    } catch (IOException e) {\n                        throw new FileConnectorException(\n                                CommonErrorCodeDeprecated.FLUSH_DATA_FAILED,\n                                String.format(\"Flush data to this file [%s] failed\", key),\n                                e);\n                    } finally {\n                        try {\n                            value.close();\n                        } catch (IOException e) {\n                            log.warn(\"Close file output stream {} failed\", key, e);\n                        }\n                    }\n                    needMoveFiles.put(key, getTargetLocation(key));\n                });\n        beingWrittenOutputStream.clear();\n        isFirstWrite.clear();\n    }\n\n    @Override\n    public FSDataOutputStream getOrCreateOutputStream(@NonNull String filePath) {","sourceCodeStart":89,"sourceCodeEnd":125,"githubUrl":"https://github.com/apache/seatunnel/blob/cf67b549a7a6c35fa0beb12d83c62892427ea919/seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/sink/writer/CanalJsonWriteStrategy.java#L89-L125","documentation":"CanalJsonWriteStrategy's finishAndCloseFile flushes each open output stream before closing; a failed flush throws FileConnectorException with code FLUSH_DATA_FAILED identifying the file. Unflushed Canal JSON rows for that file would be lost, so the sink fails the task explicitly.","triggerScenarios":"finishAndCloseFile is called at task end (commit/prepareCommit) and value.flush() on an FSDataOutputStream throws IOException — lost connection to HDFS/S3/OSS, disk full, or handle invalidated.","commonSituations":"Object-store endpoint throttling or outage during checkpoint commit; local disk full on the worker; long-lived stream hitting idle-timeout on the storage backend.","solutions":["Check the wrapped IOException cause and the storage backend's health at commit time","Free disk space / raise quotas on the target filesystem and rerun the job","Enable client-side retries/timeouts for the object store or HDFS, and shorten checkpoint intervals to reduce buffered data per flush"],"exampleFix":"null","handlingStrategy":"try-catch","validationCode":"// pre-flight storage check before long Canal-JSON writes\ntry (FSDataOutputStream probe = fs.create(new Path(sinkPath, \".probe\"))) {\n  probe.writeUTF(\"ok\"); probe.hflush();\n  fs.delete(new Path(sinkPath, \".probe\"), true);\n}","typeGuard":null,"tryCatchPattern":"try { strategy.finishAndCloseFile(); } catch (FileConnectorException e) {\n  if (CommonErrorCode.FLUSH_DATA_FAILED.equals(e.getErrorCode())) {\n    log.error(\"Canal JSON flush failed for {}; cause: {}\", e.getMessage(), e.getCause());\n    // let checkpoint/task retry recover; verify storage health\n  }\n}","preventionTips":["Watch for object-store throttling at commit time and add client retries","Ensure sufficient free space on the target filesystem","Reduce time between flushes by tuning checkpoint intervals for high-throughput CDC streams"],"tags":["file-sink","canal-json","flush","io"],"backgroundTag":"file-write-failed","analyzedSha":"cf67b549a7a6c35fa0beb12d83c62892427ea919","analyzedAt":"2026-09-10T21:44:55.265Z","contentChangedAt":"2026-09-10T21:44:55.265Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}