apache/beam · critical · IllegalStateException

Unable to create RecordWriter object:

Error message

Unable to create RecordWriter object: 

What it means

The sink initializes a RecordWriter via OutputFormat.getRecordWriter(taskAttemptContext). If that call throws an IOException or InterruptedException, this IllegalStateException is thrown, wrapping the cause.

Solutions

1. Ensure the output path does not already exist (or configure a committer/format that overwrites)
2. Verify write permissions on the output filesystem
3. Check the chained cause for the concrete IOException (codec, path, config) and fix the job configuration
4. Validate the configured OutputFormat class and its compression settings

Example fix

// before
p.apply("Write", HadoopIO.write().to("hdfs://out/result")); // out/result already exists
// after
deleteExistingOutputDir("hdfs://out/result"); // or write to a new unique path
p.apply("Write", HadoopIO.write().to("hdfs://out/result-" + runId));

Path out = new Path(outputDir);
FileSystem fs = out.getFileSystem(conf);
if (fs.exists(out)) throw new IllegalStateException("Output dir exists: " + out);
if (!fs.mkdirs(out.getParent())) throw new IllegalStateException("Cannot create parent: " + out.getParent());

try {
  pipeline.run().waitUntilFinish();
} catch (Exception e) {
  if (hasCauseMessage(e, "Unable to create RecordWriter object")) {
    throw new IllegalStateException("Check output path, permissions, and output format config", e);
  }
  throw e;
}

When it happens

Trigger: OutputFormat.getRecordWriter() fails due to unwritable output path, existing output directory (with FileOutputFormat's fail-if-exists), missing compression codec, or interruption while creating the writer.

Common situations: Output directory already exists; missing HDFS write permissions; misconfigured output format/compression settings; Hadoop version mismatch in the OutputFormat class.

Understand the failure class

Background: "failed to write file", "Could not save figure", "Error saving remote file" — file write failed: causes and fixes across languages and libraries — this error's family across 38 libraries.

Related errors

• Unable to abort task
• AUTO is not supported for writing
• Cannot create reader as source is not split yet.
• Cannot output with timestamp
• Cannot provide because does not implement the interface

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/50332143dc43d723. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/io/hadoop-format/src/main/java/org/apache/beam/sdk/io/hadoop/format/HadoopFormatIO.java:1572

      try {
        outputCommitter.abortTask(taskAttemptContext);
      } catch (IOException e) {
        throw new IllegalStateException(
            String.format("Unable to abort task %s of job %s", getTaskId(), getJobId()));
      }
    }

    private RecordWriter<KeyT, ValueT> initRecordWriter(
        OutputFormat<KeyT, ValueT> outputFormatObj, TaskAttemptContext taskAttemptContext)
        throws IllegalStateException {
      try {
        LOG.info(
            "Creating new RecordWriter for task {} of Job with id {}.",
            taskAttemptContext.getTaskAttemptID().getTaskID().getId(),
            taskAttemptContext.getJobID().getJtIdentifier());
        return outputFormatObj.getRecordWriter(taskAttemptContext);
      } catch (InterruptedException | IOException e) {
        throw new IllegalStateException("Unable to create RecordWriter object: ", e);
      }
    }

    private static OutputCommitter initOutputCommitter(
        OutputFormat<?, ?> outputFormatObj,
        Configuration conf,
        TaskAttemptContext taskAttemptContext)
        throws IllegalStateException {
      OutputCommitter outputCommitter;
      try {
        outputCommitter = outputFormatObj.getOutputCommitter(taskAttemptContext);
        if (outputCommitter != null) {
          outputCommitter.setupJob(new JobContextImpl(conf, taskAttemptContext.getJobID()));
        }
      } catch (Exception e) {
        throw new IllegalStateException("Unable to create OutputCommitter object: ", e);
      }

View on GitHub (pinned to 12126d8942)