apache/seatunnel · error · FileConnectorException

FLUSH_DATA_FAILED

FLUSH_DATA_FAILED

Error message

Flush data to this file [%s] failed

What it means

At the end of a sink task, finishAndCloseFile flushes and closes every open output stream. If flush() throws IOException for a given file, BinaryWriteStrategy wraps it in a FileConnectorException with code FLUSH_DATA_FAILED naming the offending file. Data buffered for that file was not durably written, so the task fails rather than silently losing bytes.

Source

Thrown at seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/sink/writer/BinaryWriteStrategy.java:161

        return fsDataOutputStream;
    }

    @Override
    public void applySchemaChange(SchemaChangeEvent event) {
        throw new FileConnectorException(
                FileConnectorErrorCode.FORMAT_NOT_SUPPORT,
                "BinaryWriteStrategy does not support schema evolution. "
                        + "Binary format requires a fixed schema.");
    }

    @Override
    public void finishAndCloseFile() {
        beingWrittenOutputStream.forEach(
                (key, value) -> {
                    try {
                        value.flush();
                    } catch (IOException e) {
                        throw new FileConnectorException(
                                CommonErrorCodeDeprecated.FLUSH_DATA_FAILED,
                                String.format("Flush data to this file [%s] failed", key),
                                e);
                    } finally {
                        try {
                            value.close();
                        } catch (IOException e) {
                            log.error("error when close output stream {}", key, e);
                        }
                    }
                    needMoveFiles.put(key, getTargetLocation(key));
                });
        beingWrittenOutputStream.clear();
        partIndexMap.clear();
    }
}

View on GitHub (pinned to cf67b549a7)

Solutions

  1. Inspect the wrapped IOException cause for the storage-level root cause (connectivity, space, quota)
  2. Verify free space/quotas on the target filesystem and retry the job
  3. Add storage retry/timeout tuning (e.g. S3 client retries, HDFS client timeouts) for flaky endpoints

Example fix

null
Defensive patterns

Strategy: try-catch

Validate before calling

// pre-flight: ensure the target filesystem accepts writes and has space
try (FSDataOutputStream probe = fs.create(new Path(sinkPath, ".probe"))) {
  probe.write(new byte[1]); probe.hflush();
  fs.delete(new Path(sinkPath, ".probe"), true);
}

Try / catch

try { strategy.finishAndCloseFile(); } catch (FileConnectorException e) {
  if (CommonErrorCode.FLUSH_DATA_FAILED.equals(e.getErrorCode())) {
    log.error("Flush failed for output file; storage may be unavailable: {}", e.getCause());
    // rely on engine retry/checkpoint to re-run the task
  }
}

Prevention

When it happens

Trigger: finishAndCloseFile iterates beingWrittenOutputStream and the underlying Hadoop FSDataOutputStream.flush() throws — storage endpoint outage, disk full, HDFS/S3 connection dropped mid-write.

Common situations: S3/HDFS network interruption at task completion; quota or disk-space exhaustion on the target filesystem; stale file handles after a long-running task.

Understand the failure class

Background: "failed to write file", "Could not save figure", "Error saving remote file" — file write failed: causes and fixes across languages and libraries — this error's family across 38 libraries.

Related errors

AI-assisted analysis of apache/seatunnel@cf67b549a7 (2026-09-10). Data as JSON: /api/errors/b0f42e21f026402c. Report an issue: GitHub.