apache/flink · error · IOException

Write record failed

Error message

Write record failed

What it means

Thrown by OutputFormatBase.checkAsyncErrors as an IOException wrapping the first asynchronous write error captured in an AtomicReference. The base class records failures from send()/flush() completions and re-throws them synchronously on the next checkAsyncErrors call (in close(), and around flush()). This turns silent async failures into visible exceptions at lifecycle boundaries.

Source

Thrown at flink-core/src/main/java/org/apache/flink/api/common/io/OutputFormatBase.java:143

                    if (throwable == null) {
                        callback.onSuccess(result);
                    } else {
                        callback.onFailure(throwable);
                    }
                });
    }

    /**
     * Send the actual record for writing.
     *
     * @return a CompletionStage that represents the writing task.
     */
    protected abstract CompletionStage<V> send(OUT record);

    private void checkAsyncErrors() throws IOException {
        final Throwable currentError = throwable.getAndSet(null);
        if (currentError != null) {
            throw new IOException("Write record failed", currentError);
        }
    }

    /** Close the format waiting for pending writes and reports errors. */
    @Override
    public final void close() throws IOException {
        checkAsyncErrors();
        flush();
        checkAsyncErrors();
        postClose();
    }

    /**
     * Tear down the OutputFormat. This method is called at the end of {@link
     * OutputFormatBase#close()}.
     */
    protected void postClose() {}

View on GitHub (pinned to 2f3c205e92)

Solutions

1. Inspect the wrapped cause in the IOException to find the real failure (network, auth, schema, disk).
2. Make send() retry transient errors (with backoff) so only permanent failures propagate to checkAsyncErrors.
3. Ensure flush() is called periodically so errors surface before the final close, and add error-handling/retry around close().
4. Add monitoring on the async write path so failures are visible immediately rather than only at close.

Example fix

// before: send fails and error only surfaces in close()
protected CompletionStage<Void> send(OUT r) { return client.write(r); }
// after: retry transient failures inside send
protected CompletionStage<Void> send(OUT r) {
  return client.write(r).exceptionallyCompose(
      e -> isTransient(e) ? retryWithBackoff(() -> client.write(r)) : failedFuture(e));
}

// Before close(), drain pending writes and surface errors early
format.flush();              // triggers checkAsyncErrors internally
// in a periodic loop while writing:
format.flush();              // surfaces async failures before the final close

try {
    format.close();
} catch (IOException e) {
    Throwable cause = e.getCause();
    if (cause instanceof TransientException) {
        // retry the failed batch if the format supports replay
        retryBatch();
    } else {
        throw e; // permanent failure
    }
}

When it happens

Trigger: A subclass of OutputFormatBase implements send(record) returning a CompletionStage that completes exceptionally (e.g. network error, rejected record, serialization failure). The error is stored; the next checkAsyncErrors (called from close, or between writes) wraps it in IOException('Write record failed', cause) and rethrows.

Common situations: Sink/OutputFormat writing to a database, queue, or HTTP endpoint where the async send fails: connection drop, auth expiry, schema rejection, timeout, full disk on flush. Errors may lag behind the records that caused them, surfacing only at close().

Related errors

• Error while waiting for job to be initialized
• Could not build the program from JAR file: {}
• Could not cancel job {}.
• Missing JobID. Specify a JobID to cancel a job.
• Failed to dispose the savepoint '{}'.

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/72236c6d30a845b5. Report an issue: GitHub.