apache/seatunnel · error · FileConnectorException

FLUSH_DATA_FAILED

FLUSH_DATA_FAILED

Error message

Flush data to this file [%s] failed

What it means

Thrown by JsonWriteStrategy.finishAndCloseFile when flushing buffered JSON output to the underlying file stream fails with an IOException. Raised at finalization (close/checkpoint), meaning the JSON output file may be incomplete. The original IOException is attached as the cause.

Source

Thrown at seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/sink/writer/JsonWriteStrategy.java:99

            if (isFirstWrite.get(filePath)) {
                isFirstWrite.put(filePath, false);
            } else {
                fsDataOutputStream.write(rowDelimiter);
            }
            fsDataOutputStream.write(rowBytes);
        } catch (IOException e) {
            throw CommonError.fileOperationFailed("JsonFile", "write", filePath, e);
        }
    }

    @Override
    public void finishAndCloseFile() {
        beingWrittenOutputStream.forEach(
                (key, value) -> {
                    try {
                        value.flush();
                    } catch (IOException e) {
                        throw new FileConnectorException(
                                CommonErrorCodeDeprecated.FLUSH_DATA_FAILED,
                                String.format("Flush data to this file [%s] failed", key),
                                e);
                    } finally {
                        try {
                            value.close();
                        } catch (IOException e) {
                            log.warn("Close file output stream {} failed", key, e);
                        }
                    }
                    needMoveFiles.put(key, getTargetLocation(key));
                });
        beingWrittenOutputStream.clear();
        isFirstWrite.clear();
    }

    @Override
    public FSDataOutputStream getOrCreateOutputStream(@NonNull String filePath) {

View on GitHub (pinned to cf67b549a7)

Solutions

  1. Read e.getCause() to find the root storage problem (ENOSPC, permission, timeout) and remediate it
  2. Check target path availability and quota; clean up stale output directories
  3. Ensure exclusive access to the output path (one writer per file); use job-specific temp dirs
  4. Resubmit the job after fixing storage; most flush failures are transient

Example fix

// before
try { Files.write(path, bytes); } catch (IOException ignored) { }
// after
try { out.flush(); } catch (IOException e) {
    throw new FileConnectorException(CommonErrorCodeDeprecated.FLUSH_DATA_FAILED,
        String.format("Flush data to this file [%s] failed", key), e);
}
Defensive patterns

Strategy: try-catch

Validate before calling

// verify output dir exists and is writable
FileSystem fs = path.getFileSystem(conf);
if (!fs.exists(outDir) || !fs.getFileStatus(outDir).getPermission().getAction().contains("w")) { throw new IllegalStateException("output dir not writable"); }

Try / catch

try {
    job.execute();
} catch (FileConnectorException e) {
    if (CommonErrorCodeDeprecated.FLUSH_DATA_FAILED.equals(e.getErrorCode())) {
        // check e.getCause(): ENOSPC / AccessControl / timeout
        // clean stale outputs, fix storage, retry
    }
}

Prevention

When it happens

Trigger: value.flush() throws IOException while closing JSON sink files — full disk, HDFS/S3 write error, stream closed early, or file deleted externally.

Common situations: Cluster storage quota exhaustion; transient network outage to object store; multiple jobs writing the same output file; filesystem unmounted mid-job.

Understand the failure class

Background: "failed to write file", "Could not save figure", "Error saving remote file" — file write failed: causes and fixes across languages and libraries — this error's family across 38 libraries.

Related errors

AI-assisted analysis of apache/seatunnel@cf67b549a7 (2026-09-10). Data as JSON: /api/errors/f8f52b92066e7e77. Report an issue: GitHub.