Failed to persist failure sample to '{}'; disabling failure-

Error message

Failed to persist failure sample to '{}'; disabling failure-data persistence. cause={}

What it means

Logged by BatchBuffer.writeFailureSample when persisting a failed-record sample to the configured failure-data file throws an IOException. The writer is permanently disabled (failureWriterDisabled = true) for the rest of the task, and only the exception message is logged as WARN. Future failed records will still be skipped but no longer saved to disk.

Source

Thrown at seatunnel-connectors-v2/connector-hugegraph/src/main/java/org/apache/seatunnel/connectors/seatunnel/hugegraph/buffer/BatchBuffer.java:367

                if (!dir.exists() && !dir.mkdirs() && !dir.exists()) {
                    throw new IOException("Failed to create failure data directory: " + dir);
                }
                File file =
                        new File(dir, "hugegraph-sink-failures-subtask-" + subtaskIndex + ".log");
                failureWriter =
                        new BufferedWriter(
                                new OutputStreamWriter(
                                        new FileOutputStream(file, true), StandardCharsets.UTF_8));
                LOG.info("Persisting skipped-record failure samples to {}", file.getAbsolutePath());
            }
            failureWriter.write(formatFailureSample(envelope, failure));
            failureWriter.newLine();
            // Flush per record: failures are rare and losing samples on an abrupt crash defeats
            // their purpose.
            failureWriter.flush();
        } catch (IOException e) {
            failureWriterDisabled = true;
            LOG.warn(
                    "Failed to persist failure sample to '{}'; disabling failure-data persistence. cause={}",
                    failureDataPath,
                    e.getMessage());
        }
    }

    /**
     * One-line, tab-delimited failure sample. Newlines are stripped to keep one record per line.
     */
    static String formatFailureSample(GraphElementEnvelope envelope, Exception failure) {
        GraphElement element = envelope.getElement();
        String line =
                String.format(
                        "mapping=%s\ttype=%s\tid=%s\tlabel=%s\tproperties=%s\terror=%s",
                        envelope.getMappingLabel(),
                        envelope.getElementType(),
                        element == null ? null : element.id(),
                        element == null ? null : element.label(),

View on GitHub (pinned to cf67b549a7)

Solutions

Verify the failureDataPath directory exists and is writable by the SeaTunnel worker process user
Check disk space on the worker node (df -h) and free space if full
Correct the failure-data path in the sink configuration to a valid, writable location
Restart the job after fixing the path; the writer stays disabled until the task is recreated

Example fix

// before
"failure-data-path" = "/read-only-mount/failures.log"
// after
"failure-data-path" = "/tmp/seatunnel/failures.log"  // writable directory, pre-created
// mkdir -p /tmp/seatunnel && chmod u+w /tmp/seatunnel

Defensive patterns

Strategy: validation

Validate before calling

// Before submitting the job
Path dir = Paths.get(failureDataPath).getParent();
if (!Files.isDirectory(dir) || !Files.isWritable(dir)) {
    throw new IllegalStateException("failure-data path not writable: " + dir);
}
if (dir.toFile().getUsableSpace() < minRequiredBytes) {
    throw new IllegalStateException("insufficient disk space for failure samples");
}

Prevention

Point failureDataPath at a writable local directory on every worker
Pre-create the directory in deployment scripts (mkdir -p with correct ownership)
Monitor worker disk usage; keep headroom for failure samples
Avoid read-only container filesystems or read-only NFS mounts for this path

When it happens

Trigger: Writing/flushing a failure record to failureDataPath fails: the parent directory does not exist or is not writable, the disk is full, the path is invalid, or the file handle was closed/deleted while the job runs. Called from fallbackInsertSingly after record failures.

Common situations: Misconfigured failure-data path (read-only mount, container without write permission to the configured directory), disk-full on worker nodes, or the output directory being cleaned up by an external process during a long-running job.

Understand the failure class

Background: "failed to write file", "Could not save figure", "Error saving remote file" — file write failed: causes and fixes across languages and libraries — this error's family across 38 libraries.

Related errors

AI-assisted analysis of apache/seatunnel@cf67b549a7 (2026-09-10). Data as JSON: /api/errors/a670de58ec4c39ce. Report an issue: GitHub.