apache/seatunnel · error · FileConnectorException
FLUSH_DATA_FAILED
FLUSH_DATA_FAILED
Error message
Flush data to this file [%s] failed
What it means
Thrown by DebeziumJsonWriteStrategy.finishAndCloseFile when flushing the output stream of a Debezium-JSON formatted file fails with an IOException during finalization. The buffered JSON records could not be committed to storage, so the produced file may be truncated. The underlying IOException is preserved as the cause.
Source
Thrown at seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/sink/writer/DebeziumJsonWriteStrategy.java:107
if (isFirstWrite.get(filePath)) {
isFirstWrite.put(filePath, false);
} else {
fsDataOutputStream.write(rowDelimiter);
}
fsDataOutputStream.write(rowBytes);
} catch (IOException e) {
throw CommonError.fileOperationFailed("DebeziumJsonFile", "write", filePath, e);
}
}
@Override
public void finishAndCloseFile() {
beingWrittenOutputStream.forEach(
(key, value) -> {
try {
value.flush();
} catch (IOException e) {
throw new FileConnectorException(
CommonErrorCodeDeprecated.FLUSH_DATA_FAILED,
String.format("Flush data to this file [%s] failed", key),
e);
} finally {
try {
value.close();
} catch (IOException e) {
log.warn("Close file output stream {} failed", key, e);
}
}
needMoveFiles.put(key, getTargetLocation(key));
});
beingWrittenOutputStream.clear();
isFirstWrite.clear();
}
@Override
public FSDataOutputStream getOrCreateOutputStream(@NonNull String filePath) {View on GitHub (pinned to cf67b549a7)
Solutions
- Inspect the chained cause to identify the storage-level failure and fix it (space, credentials, connectivity)
- Validate the target filesystem is writable before submitting the job (write/read a probe file)
- Avoid concurrent jobs writing the same output directory; use distinct transaction paths per job
- Retry the pipeline; transient storage failures during finalization frequently succeed on rerun
Example fix
// before
value.flush(); // unchecked IOException propagates without context
// after
try { value.flush(); } catch (IOException e) {
throw new FileConnectorException(CommonErrorCodeDeprecated.FLUSH_DATA_FAILED,
String.format("Flush data to this file [%s] failed", key), e);
} Defensive patterns
Strategy: try-catch
Validate before calling
// pre-check filesystem writability assert fs.createNewFile(new Path(outputDir, ".write_check"));
Try / catch
try {
job.execute();
} catch (FileConnectorException e) {
if (CommonErrorCodeDeprecated.FLUSH_DATA_FAILED.equals(e.getErrorCode())) {
log.error("Flush failed: ", e.getCause());
// remediate storage then resubmit
}
} Prevention
- Monitor storage quota/credentials validity for long-running jobs
- Avoid concurrent writers to the same output directory
- Use job-unique transaction/temp directories
- Catch FileConnectorException and inspect getCause() for root cause
When it happens
Trigger: value.flush() on FSDataOutputStream in beingWrittenOutputStream throws IOException at sink close/checkpoint — disk full, remote storage (HDFS/S3) unreachable, or stream already closed.
Common situations: HDFS DataNode failure or replication timeout during job finish; S3/OSS throttling or expired credentials; disk quota exceeded on local checkpoint dir; concurrent writers to the same path.
Understand the failure class
Background: "failed to write file", "Could not save figure", "Error saving remote file" — file write failed: causes and fixes across languages and libraries — this error's family across 38 libraries.
Related errors
AI-assisted analysis of apache/seatunnel@cf67b549a7 (2026-09-10).
Data as JSON: /api/errors/e083eb6b607e5eda.
Report an issue: GitHub.