{"record":{"id":"3673359f500b2c5f","repo":"apache/beam","slug":"unable-to-commit-job","errorCode":null,"errorMessage":"Unable to commit job.","messagePattern":"Unable to commit job\\.","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"critical","filePath":"sdks/java/io/hadoop-format/src/main/java/org/apache/beam/sdk/io/hadoop/format/HadoopFormatIO.java","lineNumber":1828,"sourceCode":"    }\n\n    /**\n     * Commits whole write job.\n     *\n     * @param config hadoop config\n     */\n    private void cleanupJob(Configuration config) {\n\n      externalSynchronization.releaseJobIdLock(config);\n\n      JobID jobID = HadoopFormats.getJobId(config);\n      TaskAttemptContext cleanupTaskContext = HadoopFormats.createCleanupTaskContext(config, jobID);\n      OutputFormat<?, ?> outputFormat = HadoopFormats.createOutputFormatFromConfig(config);\n      try {\n        OutputCommitter outputCommitter = outputFormat.getOutputCommitter(cleanupTaskContext);\n        outputCommitter.commitJob(cleanupTaskContext);\n      } catch (Exception e) {\n        throw new RuntimeException(\"Unable to commit job.\", e);\n      }\n    }\n  }\n\n  /**\n   * Assigns {@link TaskID#getId()} to the given pair of key and value. {@link TaskID} is later used\n   * for writing the pair to hadoop file.\n   *\n   * @param <KeyT> Type of key\n   * @param <ValueT> Type of value\n   */\n  private static class AssignTaskFn<KeyT, ValueT>\n      extends DoFn<KV<KeyT, ValueT>, KV<Integer, KV<KeyT, ValueT>>> {\n\n    private final PCollectionView<Configuration> configView;\n\n    // Transient properties because they are used only for one bundle\n    /** Cache of created TaskIDs for given bundle. */","sourceCodeStart":1810,"sourceCodeEnd":1846,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/java/io/hadoop-format/src/main/java/org/apache/beam/sdk/io/hadoop/format/HadoopFormatIO.java#L1810-L1846","documentation":"At finalization, HadoopFormatIO creates a cleanup TaskAttemptContext, rebuilds the OutputFormat from config, gets its OutputCommitter, and calls commitJob. Any failure during commitJob is rethrown as RuntimeException('Unable to commit job.').","triggerScenarios":"Finalization of the HadoopFormatIO write when outputFormat.getOutputCommitter(cleanupTaskContext) returns null (NPE on commitJob) or commitJob throws (missing task outputs, corrupted _temporary dir, HDFS errors).","commonSituations":"Some tasks failed and never wrote their outputs; _temporary directory was manually deleted; HDFS outage during commit; incompatible OutputCommitter (e.g. one that does not implement commitJob for the Hadoop version).","solutions":["Inspect the wrapped cause; if it's MissingFiles/no output, check whether any writer tasks failed and re-run the pipeline.","Ensure all write tasks succeeded before finalization (the committer requires every task attempt committed).","Verify the OutputCommitter class in configuration is compatible with your Hadoop version and supports JobCommitter API.","Check HDFS health/permissions on the output and _temporary directories."],"exampleFix":"// before\nconf.set(\"mapreduce.outputcommitter.factory.scheme\", null); // wrong/missing committer\n// after\nconf.setClass(\"mapreduce.outputcommitter.factory.scheme\",\n    org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.class,\n    org.apache.hadoop.mapreduce.OutputCommitter.class);","handlingStrategy":"try-catch","validationCode":"// ensure committer class supports commitJob\nClass<?> committer = conf.getClass(\"mapreduce.outputcommitter.factory.scheme\", null, OutputCommitter.class);\nif (committer == null) throw new IllegalStateException(\"no OutputCommitter configured\");","typeGuard":null,"tryCatchPattern":"try {\n  pipeline.run().waitUntilFinish();\n} catch (RuntimeException e) {\n  if (e.getMessage() != null && e.getMessage().contains(\"Unable to commit job.\")) {\n    LOG.error(\"Job commit failed; check task outputs and HDFS health: {}\", e.getCause(), e.getCause());\n    throw e; // commit failure = data not finalized; do not swallow\n  }\n  throw e;\n}","preventionTips":["Never manually delete _temporary directories under the output path during a run.","Ensure all writer tasks succeed — a failed task leads to commit failures downstream.","Use an OutputCommitter compatible with your Hadoop version (commitJob API).","Monitor HDFS NameNode/RegionServer health during the finalization phase."],"tags":["hadoop","hadoopformatio","job-commit","hdfs"],"backgroundTag":"database-write-failed","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}