apache/seatunnel · critical · FileConnectorException

FLUSH_DATA_FAILED

FLUSH_DATA_FAILED

Error message

Flush data to this file [%s] failed

What it means

TextWriteStrategy.finishAndCloseFile flushes every open output stream when the sink finishes/closes. If OutputStream.flush() throws IOException, it wraps it as FileConnectorException with FLUSH_DATA_FAILED naming the file; the stream is still closed in the finally block. This surfaces end-of-write I/O failures such as lost connectivity to storage or disk errors.

Source

Thrown at seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/sink/writer/TextWriteStrategy.java:127

            if (isFirstWrite.get(filePath)) {
                isFirstWrite.put(filePath, false);
            } else {
                fsDataOutputStream.write(rowDelimiter.getBytes(charset));
            }
            fsDataOutputStream.write(serializationSchema.serialize(safeProjectedRow(seaTunnelRow)));
        } catch (IOException e) {
            throw CommonError.fileOperationFailed("TextFile", "write", filePath, e);
        }
    }

    @Override
    public void finishAndCloseFile() {
        beingWrittenOutputStream.forEach(
                (key, value) -> {
                    try {
                        value.flush();
                    } catch (IOException e) {
                        throw new FileConnectorException(
                                CommonErrorCodeDeprecated.FLUSH_DATA_FAILED,
                                String.format("Flush data to this file [%s] failed", key),
                                e);
                    } finally {
                        try {
                            value.close();
                        } catch (IOException e) {
                            log.error("error when close output stream {}", key, e);
                        }
                    }
                    needMoveFiles.put(key, getTargetLocation(key));
                });
        beingWrittenOutputStream.clear();
        isFirstWrite.clear();
    }

    @Override
    public FSDataOutputStream getOrCreateOutputStream(@NonNull String filePath) {

View on GitHub (pinned to cf67b549a7)

Solutions

  1. Inspect the wrapped cause IOException to identify the storage-level root cause
  2. Check disk space/quota and storage (HDFS/S3) availability on the worker at failure time
  3. Enable sink checkpoint/rolling so partial data is flushed periodically rather than only at close
  4. Re-run the job after storage recovers; make the sink idempotent (overwrite/unique file names) for safe retries
Defensive patterns

Strategy: try-catch

Try / catch

try { strategy.finishAndCloseFile(); } catch (FileConnectorException e) { if (e.getErrorCode() == FLUSH_DATA_FAILED) { log(e.getCause()); /* inspect storage then re-run */ } }

Prevention

When it happens

Trigger: Job finish/checkpoint close on a text-format file sink while flush() on a buffered output stream fails — remote FS disconnection, disk full, HDFS lease issues, or file deleted underneath the writer.

Common situations: S3/HDFS transient outage at job completion; NFS mount dropped; disk quota exceeded mid-write; worker node storage failure; writing to a path whose parent was removed concurrently.

Understand the failure class

Background: "failed to write file", "Could not save figure", "Error saving remote file" — file write failed: causes and fixes across languages and libraries — this error's family across 38 libraries.

Related errors

AI-assisted analysis of apache/seatunnel@cf67b549a7 (2026-09-10). Data as JSON: /api/errors/b07c43e563f4caf8. Report an issue: GitHub.