apache/flink · error · IOException

Could not create RecordWriter.

Error message

Could not create RecordWriter.

What it means

Wraps an InterruptedException raised by mapreduceOutputFormat.getRecordWriter(context) inside open() while the task sets up its writer. Flink re-throws it as an IOException per the OutputFormat contract. It reflects writer creation being interrupted, typically a task-lifecycle event, not a configuration error.

Source

Thrown at flink-connectors/flink-hadoop-compatibility/src/main/java/org/apache/flink/api/java/hadoop/mapreduce/HadoopOutputFormatBase.java:164

            this.context.getCredentials().addAll(this.credentials);
            Credentials currentUserCreds =
                    getCredentialsFromUGI(UserGroupInformation.getCurrentUser());
            if (currentUserCreds != null) {
                this.context.getCredentials().addAll(currentUserCreds);
            }

            // compatible for hadoop 2.2.0, the temporary output directory is different from hadoop
            // 1.2.1
            if (outputCommitter instanceof FileOutputCommitter) {
                this.configuration.set(
                        "mapreduce.task.output.dir",
                        ((FileOutputCommitter) this.outputCommitter).getWorkPath().toString());
            }

            try {
                this.recordWriter = this.mapreduceOutputFormat.getRecordWriter(this.context);
            } catch (InterruptedException e) {
                throw new IOException("Could not create RecordWriter.", e);
            }
        }
    }

    /**
     * commit the task by moving the output file out from the temporary directory.
     *
     * @throws java.io.IOException
     */
    @Override
    public void close() throws IOException {

        // enforce sequential close() calls
        synchronized (CLOSE_MUTEX) {
            try {
                this.recordWriter.close(this.context);
            } catch (InterruptedException e) {
                throw new IOException("Could not close RecordReader.", e);

View on GitHub (pinned to 2f3c205e92)

Solutions

1. Inspect the wrapped InterruptedException in the logs to find the interrupt source (cancellation, failover).
2. Verify the output directory exists/ is writable and the filesystem (HDFS/S3) is reachable from the TaskManager.
3. Ensure output-committer configuration (mapreduce.task.output.dir) is correct so getRecordWriter does not block.
4. Treat as expected if the job was intentionally cancelled.

try {
    hadoopOutputFormat.open(initContext);
} catch (IOException e) {
    if (e.getCause() instanceof InterruptedException) {
        LOG.warn("RecordWriter creation interrupted; check task cancellation", e);
    } else {
        throw e;
    }
}

When it happens

Trigger: Produced in HadoopOutputFormatBase.open(context) when this.mapreduceOutputFormat.getRecordWriter(this.context) throws InterruptedException — e.g. the task was cancelled/failing during writer setup, the underlying filesystem open was interrupted, or a committer/lock blocked and was interrupted.

Common situations: Task cancelled while opening the sink; output filesystem unreachable or slow during writer creation; failover interrupting setup; a custom OutputFormat whose getRecordWriter blocks and mishandles interrupts.

Related errors

• Could not write Record.
• Could not close RecordReader.
• Could not get KeyValue pair.
• Could not get Splits.
• Could not create RecordReader.

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/93fc37907517ce1b. Report an issue: GitHub.