apache/seatunnel · error · FileConnectorException

FLUSH_DATA_FAILED

FLUSH_DATA_FAILED

Error message

Flush data to this file [%s] failed

What it means

Thrown by CsvWriteStrategy.finishAndCloseFile when flushing buffered bytes to the underlying Hadoop FSDataOutputStream for a CSV output file fails with an IOException. This happens at file finalization, so the file being written is likely incomplete or corrupt. The original IOException is chained as the cause.

Source

Thrown at seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/sink/writer/CsvWriteStrategy.java:133

            if (isFirstWrite.get(filePath)) {
                isFirstWrite.put(filePath, false);
            } else {
                fsDataOutputStream.write(rowDelimiter.getBytes(charset));
            }
            fsDataOutputStream.write(serializationSchema.serialize(safeProjectedRow(seaTunnelRow)));
        } catch (IOException e) {
            throw CommonError.fileOperationFailed("CsvFile", "write", filePath, e);
        }
    }

    @Override
    public void finishAndCloseFile() {
        beingWrittenOutputStream.forEach(
                (key, value) -> {
                    try {
                        value.flush();
                    } catch (IOException e) {
                        throw new FileConnectorException(
                                CommonErrorCodeDeprecated.FLUSH_DATA_FAILED,
                                String.format("Flush data to this file [%s] failed", key),
                                e);
                    } finally {
                        try {
                            value.close();
                        } catch (IOException e) {
                            log.error("error when close output stream {}", key, e);
                        }
                    }
                    needMoveFiles.put(key, getTargetLocation(key));
                });
        beingWrittenOutputStream.clear();
        isFirstWrite.clear();
    }

    @Override
    public FSDataOutputStream getOrCreateOutputStream(@NonNull String filePath) {

View on GitHub (pinned to cf67b549a7)

Solutions

  1. Check the chained cause (e.getCause()) for the real IOException (disk full, permission, connection) and free space or fix storage access
  2. Verify connectivity and quota on the target filesystem (hdfs dfs -put a test file, check S3/OSS credentials and bucket quota)
  3. Ensure only one job writes to the same output path; enable overwrite or use unique transaction/step directories
  4. Retry the job — flush failures on remote FS are often transient

Example fix

// before
try { value.flush(); } catch (IOException e) { e.printStackTrace(); }
// after
try { value.flush(); } catch (IOException e) {
    throw new FileConnectorException(CommonErrorCodeDeprecated.FLUSH_DATA_FAILED,
        "Flush data to this file [" + key + "] failed", e);
}
Defensive patterns

Strategy: try-catch

Validate before calling

// before job: verify storage writable
try (FSDataOutputStream probe = fs.create(new Path(outputDir, ".probe"))) { probe.write(1); probe.flush(); }

Try / catch

try {
    job.execute();
} catch (FileConnectorException e) {
    if (CommonErrorCodeDeprecated.FLUSH_DATA_FAILED.equals(e.getErrorCode())) {
        Throwable root = e.getCause(); // inspect disk-full / connectivity
        // free space or fix FS access, then retry
    }
}

Prevention

When it happens

Trigger: value.flush() throws IOException while iterating beingWrittenOutputStream during finishAndCloseFile — typically disk full, HDFS/DataNode write failure, or the stream/file was already closed or deleted.

Common situations: Target filesystem (HDFS/S3/OSS) out of quota or disk full; transient network partition to remote storage during checkpoint/close; permissions revoked mid-write; writing to a file deleted by another job.

Understand the failure class

Background: "failed to write file", "Could not save figure", "Error saving remote file" — file write failed: causes and fixes across languages and libraries — this error's family across 38 libraries.

Related errors

AI-assisted analysis of apache/seatunnel@cf67b549a7 (2026-09-10). Data as JSON: /api/errors/5a1b189b56409f0d. Report an issue: GitHub.