What causes "Unable to create OutputCommitter object: " in apache/beam?

Verify the OutputFormat class in configuration is correct, instantiable, and matches your sink (createOutputFormatFromConfig).

apache/beam · error · IllegalStateException

Unable to create OutputCommitter object:

Error message

Unable to create OutputCommitter object:

What it means

HadoopFormatIO's Write support needs an Hadoop OutputCommitter to set up the job. It obtains one from the configured OutputFormat via getOutputCommitter(taskAttemptContext) and calls setupJob. Any exception from the OutputFormat or committer (reflection failure, bad config, committer exception) is rethrown as IllegalStateException.

Solutions

Verify the OutputFormat class in configuration is correct, instantiable, and matches your sink (createOutputFormatFromConfig).
Ensure the Hadoop Job configuration (output dir, filesystem, Kerberos creds) is valid before running the pipeline.
Delete/ensure the output directory does not exist or the committer is allowed to set it up.
Inspect the wrapped cause 'e' in the exception for the real Hadoop error.

Example fix

// before
conf.set("mapreduce.outputformat.class", "org.apache.hadoop.mapreduce.lib.output.TextOutputFormat");
// after (with explicit output dir set up)
conf.set("mapreduce.outputformat.class", "org.apache.hadoop.mapreduce.lib.output.TextOutputFormat");
FileOutputFormat.setOutputPath(conf, new Path("/out/" + UUID.randomUUID()));

Defensive patterns

Strategy: try-catch

Validate before calling

// validate before running sink
String fmtClass = conf.get("mapreduce.outputformat.class");
Class<?> cls = Class.forName(fmtClass);
if (cls.isInterface() || Modifier.isAbstract(cls.getModifiers())) throw new IllegalArgumentException(fmtClass + " not instantiable");
cls.getConstructor();

Try / catch

try {
  pipeline.run().waitUntilFinish();
} catch (IllegalStateException e) {
  if (e.getMessage().contains("Unable to create OutputCommitter")) {
    Throwable cause = e.getCause();
    LOG.error("OutputCommitter setup failed: {}", cause, cause);
  }
  throw e;
}

Prevention

Always set mapreduce.outputformat.class to a concrete class with a public no-arg constructor.
Pre-create or clear the output directory and verify write permissions before launching.
Run a small test job against the same Hadoop configuration before full pipeline runs.
Log and inspect the cause chain — the real Hadoop error is always wrapped.

When it happens

Trigger: Calling HadoopFormatIO.write() sink during job setup when outputFormat.getOutputCommitter() throws (bad output format class, misconfigured job conf, null-arg issues) or outputCommitter.setupJob() throws (output directory exists/inaccessible).

Common situations: Wrong mapreduce.outputformat.class configured; HDFS output directory already exists; invalid job configuration (e.g. missing output path); incompatible Hadoop version's OutputFormat behavior; transient NameNode connectivity failure.

Understand the failure class

Background: "Invalid value" and "allowed values are" config errors: what your library rejected and how to fix it — this error's family across 41 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/b5a3e06ed4faabd4. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/io/hadoop-format/src/main/java/org/apache/beam/sdk/io/hadoop/format/HadoopFormatIO.java:1588

        return outputFormatObj.getRecordWriter(taskAttemptContext);
      } catch (InterruptedException | IOException e) {
        throw new IllegalStateException("Unable to create RecordWriter object: ", e);
      }
    }

    private static OutputCommitter initOutputCommitter(
        OutputFormat<?, ?> outputFormatObj,
        Configuration conf,
        TaskAttemptContext taskAttemptContext)
        throws IllegalStateException {
      OutputCommitter outputCommitter;
      try {
        outputCommitter = outputFormatObj.getOutputCommitter(taskAttemptContext);
        if (outputCommitter != null) {
          outputCommitter.setupJob(new JobContextImpl(conf, taskAttemptContext.getJobID()));
        }
      } catch (Exception e) {
        throw new IllegalStateException("Unable to create OutputCommitter object: ", e);
      }

      return outputCommitter;
    }

    @Override
    public String toString() {
      return "TaskContext{"
          + "jobId="
          + getJobId()
          + ", taskId="
          + getTaskId()
          + ", attemptId="
          + taskAttemptContext.getTaskAttemptID().getId()
          + '}';
    }
  }

View on GitHub (pinned to 12126d8942)