apache/seatunnel · error · FileConnectorException

FLUSH_DATA_FAILED

FLUSH_DATA_FAILED

Error message

Flush data to this file [%s] failed

What it means

Thrown by MaxWellJsonWriteStrategy.finishAndCloseFile when flushing the output stream of a Maxwell-JSON formatted file fails with an IOException during file finalization. Data buffered in memory could not be persisted, so the output file may be truncated. The triggering IOException is chained as the cause.

Source

Thrown at seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/sink/writer/MaxWellJsonWriteStrategy.java:107

            if (isFirstWrite.get(filePath)) {
                isFirstWrite.put(filePath, false);
            } else {
                fsDataOutputStream.write(rowDelimiter);
            }
            fsDataOutputStream.write(rowBytes);
        } catch (IOException e) {
            throw CommonError.fileOperationFailed("MaxWellJsonFile", "write", filePath, e);
        }
    }

    @Override
    public void finishAndCloseFile() {
        beingWrittenOutputStream.forEach(
                (key, value) -> {
                    try {
                        value.flush();
                    } catch (IOException e) {
                        throw new FileConnectorException(
                                CommonErrorCodeDeprecated.FLUSH_DATA_FAILED,
                                String.format("Flush data to this file [%s] failed", key),
                                e);
                    } finally {
                        try {
                            value.close();
                        } catch (IOException e) {
                            log.warn("Close file output stream {} failed", key, e);
                        }
                    }
                    needMoveFiles.put(key, getTargetLocation(key));
                });
        beingWrittenOutputStream.clear();
        isFirstWrite.clear();
    }

    @Override
    public FSDataOutputStream getOrCreateOutputStream(@NonNull String filePath) {

View on GitHub (pinned to cf67b549a7)

Solutions

1. Inspect the chained cause and fix the storage-level issue (space, permissions, connectivity)
2. Verify the target filesystem is healthy and writable before the job finishes (probe write)
3. Guarantee exclusive write access to output paths; isolate each job's transaction directory
4. Retry the job after remediation — finalize-time flush failures are often transient

Example fix

// before
value.flush(); // loses error context
// after
try { value.flush(); } catch (IOException e) {
    throw new FileConnectorException(CommonErrorCodeDeprecated.FLUSH_DATA_FAILED,
        String.format("Flush data to this file [%s] failed", key), e);
}

// sanity-check target storage before submit
fs.access(outDir, FsAction.WRITE); // throws if not writable

try {
    job.execute();
} catch (FileConnectorException e) {
    if (CommonErrorCodeDeprecated.FLUSH_DATA_FAILED.equals(e.getErrorCode())) {
        Throwable cause = e.getCause();
        // e.g. refresh credentials if cause is credential/token related, then retry
    }
}

When it happens

Trigger: value.flush() on the FSDataOutputStream in beingWrittenOutputStream throws IOException at sink close — disk full, HDFS/S3 outage, stream already closed, or file removed externally.

Common situations: Object-store throttling or credential expiry near the end of a long job; HDFS safe mode or DataNode loss at finalize time; local disk quota exceeded; concurrent writers on same path.

Understand the failure class

Background: "failed to write file", "Could not save figure", "Error saving remote file" — file write failed: causes and fixes across languages and libraries — this error's family across 38 libraries.

Related errors

• FLUSH_DATA_FAILED
• FLUSH_DATA_FAILED
• FLUSH_DATA_FAILED
• WRITER_OPERATION_FAILED
• FLUSH_DATA_FAILED

AI-assisted analysis of apache/seatunnel@cf67b549a7 (2026-09-10). Data as JSON: /api/errors/cdbd420394339c6d. Report an issue: GitHub.