apache/beam · critical · RuntimeException

Unable to commit job.

Error message

Unable to commit job.

What it means

At finalization, HadoopFormatIO creates a cleanup TaskAttemptContext, rebuilds the OutputFormat from config, gets its OutputCommitter, and calls commitJob. Any failure during commitJob is rethrown as RuntimeException('Unable to commit job.').

Solutions

1. Inspect the wrapped cause; if it's MissingFiles/no output, check whether any writer tasks failed and re-run the pipeline.
2. Ensure all write tasks succeeded before finalization (the committer requires every task attempt committed).
3. Verify the OutputCommitter class in configuration is compatible with your Hadoop version and supports JobCommitter API.
4. Check HDFS health/permissions on the output and _temporary directories.

Example fix

// before
conf.set("mapreduce.outputcommitter.factory.scheme", null); // wrong/missing committer
// after
conf.setClass("mapreduce.outputcommitter.factory.scheme",
    org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.class,
    org.apache.hadoop.mapreduce.OutputCommitter.class);

// ensure committer class supports commitJob
Class<?> committer = conf.getClass("mapreduce.outputcommitter.factory.scheme", null, OutputCommitter.class);
if (committer == null) throw new IllegalStateException("no OutputCommitter configured");

try {
  pipeline.run().waitUntilFinish();
} catch (RuntimeException e) {
  if (e.getMessage() != null && e.getMessage().contains("Unable to commit job.")) {
    LOG.error("Job commit failed; check task outputs and HDFS health: {}", e.getCause(), e.getCause());
    throw e; // commit failure = data not finalized; do not swallow
  }
  throw e;
}

When it happens

Trigger: Finalization of the HadoopFormatIO write when outputFormat.getOutputCommitter(cleanupTaskContext) returns null (NPE on commitJob) or commitJob throws (missing task outputs, corrupted _temporary dir, HDFS errors).

Common situations: Some tasks failed and never wrote their outputs; _temporary directory was manually deleted; HDFS outage during commit; incompatible OutputCommitter (e.g. one that does not implement commitJob for the Hadoop version).

Related errors

• Unable to setup job.
• Expected file path but received directory path
• Support for move options is not yet implemented.
• Unable to copy resource
• Unable to create OutputCommitter object:

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/3673359f500b2c5f. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/io/hadoop-format/src/main/java/org/apache/beam/sdk/io/hadoop/format/HadoopFormatIO.java:1828

    }

    /**
     * Commits whole write job.
     *
     * @param config hadoop config
     */
    private void cleanupJob(Configuration config) {

      externalSynchronization.releaseJobIdLock(config);

      JobID jobID = HadoopFormats.getJobId(config);
      TaskAttemptContext cleanupTaskContext = HadoopFormats.createCleanupTaskContext(config, jobID);
      OutputFormat<?, ?> outputFormat = HadoopFormats.createOutputFormatFromConfig(config);
      try {
        OutputCommitter outputCommitter = outputFormat.getOutputCommitter(cleanupTaskContext);
        outputCommitter.commitJob(cleanupTaskContext);
      } catch (Exception e) {
        throw new RuntimeException("Unable to commit job.", e);
      }
    }
  }

  /**
   * Assigns {@link TaskID#getId()} to the given pair of key and value. {@link TaskID} is later used
   * for writing the pair to hadoop file.
   *
   * @param <KeyT> Type of key
   * @param <ValueT> Type of value
   */
  private static class AssignTaskFn<KeyT, ValueT>
      extends DoFn<KV<KeyT, ValueT>, KV<Integer, KV<KeyT, ValueT>>> {

    private final PCollectionView<Configuration> configView;

    // Transient properties because they are used only for one bundle
    /** Cache of created TaskIDs for given bundle. */

View on GitHub (pinned to 12126d8942)