apache/seatunnel · error · FileConnectorException

FLUSH_DATA_FAILED

FLUSH_DATA_FAILED

Error message

Flush data to this file [%s] failed

What it means

CanalJsonWriteStrategy's finishAndCloseFile flushes each open output stream before closing; a failed flush throws FileConnectorException with code FLUSH_DATA_FAILED identifying the file. Unflushed Canal JSON rows for that file would be lost, so the sink fails the task explicitly.

Source

Thrown at seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/sink/writer/CanalJsonWriteStrategy.java:107

            if (isFirstWrite.get(filePath)) {
                isFirstWrite.put(filePath, false);
            } else {
                fsDataOutputStream.write(rowDelimiter);
            }
            fsDataOutputStream.write(rowBytes);
        } catch (IOException e) {
            throw CommonError.fileOperationFailed("CanalJsonFile", "write", filePath, e);
        }
    }

    @Override
    public void finishAndCloseFile() {
        beingWrittenOutputStream.forEach(
                (key, value) -> {
                    try {
                        value.flush();
                    } catch (IOException e) {
                        throw new FileConnectorException(
                                CommonErrorCodeDeprecated.FLUSH_DATA_FAILED,
                                String.format("Flush data to this file [%s] failed", key),
                                e);
                    } finally {
                        try {
                            value.close();
                        } catch (IOException e) {
                            log.warn("Close file output stream {} failed", key, e);
                        }
                    }
                    needMoveFiles.put(key, getTargetLocation(key));
                });
        beingWrittenOutputStream.clear();
        isFirstWrite.clear();
    }

    @Override
    public FSDataOutputStream getOrCreateOutputStream(@NonNull String filePath) {

View on GitHub (pinned to cf67b549a7)

Solutions

  1. Check the wrapped IOException cause and the storage backend's health at commit time
  2. Free disk space / raise quotas on the target filesystem and rerun the job
  3. Enable client-side retries/timeouts for the object store or HDFS, and shorten checkpoint intervals to reduce buffered data per flush

Example fix

null
Defensive patterns

Strategy: try-catch

Validate before calling

// pre-flight storage check before long Canal-JSON writes
try (FSDataOutputStream probe = fs.create(new Path(sinkPath, ".probe"))) {
  probe.writeUTF("ok"); probe.hflush();
  fs.delete(new Path(sinkPath, ".probe"), true);
}

Try / catch

try { strategy.finishAndCloseFile(); } catch (FileConnectorException e) {
  if (CommonErrorCode.FLUSH_DATA_FAILED.equals(e.getErrorCode())) {
    log.error("Canal JSON flush failed for {}; cause: {}", e.getMessage(), e.getCause());
    // let checkpoint/task retry recover; verify storage health
  }
}

Prevention

When it happens

Trigger: finishAndCloseFile is called at task end (commit/prepareCommit) and value.flush() on an FSDataOutputStream throws IOException — lost connection to HDFS/S3/OSS, disk full, or handle invalidated.

Common situations: Object-store endpoint throttling or outage during checkpoint commit; local disk full on the worker; long-lived stream hitting idle-timeout on the storage backend.

Understand the failure class

Background: "failed to write file", "Could not save figure", "Error saving remote file" — file write failed: causes and fixes across languages and libraries — this error's family across 38 libraries.

Related errors

AI-assisted analysis of apache/seatunnel@cf67b549a7 (2026-09-10). Data as JSON: /api/errors/f57c945bb0318205. Report an issue: GitHub.