{"record":{"id":"b5a3e06ed4faabd4","repo":"apache/beam","slug":"unable-to-create-outputcommitter-object","errorCode":null,"errorMessage":"Unable to create OutputCommitter object: ","messagePattern":"Unable to create OutputCommitter object: ","errorType":"exception","errorClass":"IllegalStateException","httpStatus":null,"severity":"error","filePath":"sdks/java/io/hadoop-format/src/main/java/org/apache/beam/sdk/io/hadoop/format/HadoopFormatIO.java","lineNumber":1588,"sourceCode":"        return outputFormatObj.getRecordWriter(taskAttemptContext);\n      } catch (InterruptedException | IOException e) {\n        throw new IllegalStateException(\"Unable to create RecordWriter object: \", e);\n      }\n    }\n\n    private static OutputCommitter initOutputCommitter(\n        OutputFormat<?, ?> outputFormatObj,\n        Configuration conf,\n        TaskAttemptContext taskAttemptContext)\n        throws IllegalStateException {\n      OutputCommitter outputCommitter;\n      try {\n        outputCommitter = outputFormatObj.getOutputCommitter(taskAttemptContext);\n        if (outputCommitter != null) {\n          outputCommitter.setupJob(new JobContextImpl(conf, taskAttemptContext.getJobID()));\n        }\n      } catch (Exception e) {\n        throw new IllegalStateException(\"Unable to create OutputCommitter object: \", e);\n      }\n\n      return outputCommitter;\n    }\n\n    @Override\n    public String toString() {\n      return \"TaskContext{\"\n          + \"jobId=\"\n          + getJobId()\n          + \", taskId=\"\n          + getTaskId()\n          + \", attemptId=\"\n          + taskAttemptContext.getTaskAttemptID().getId()\n          + '}';\n    }\n  }\n","sourceCodeStart":1570,"sourceCodeEnd":1606,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/java/io/hadoop-format/src/main/java/org/apache/beam/sdk/io/hadoop/format/HadoopFormatIO.java#L1570-L1606","documentation":"HadoopFormatIO's Write support needs an Hadoop OutputCommitter to set up the job. It obtains one from the configured OutputFormat via getOutputCommitter(taskAttemptContext) and calls setupJob. Any exception from the OutputFormat or committer (reflection failure, bad config, committer exception) is rethrown as IllegalStateException.","triggerScenarios":"Calling HadoopFormatIO.write() sink during job setup when outputFormat.getOutputCommitter() throws (bad output format class, misconfigured job conf, null-arg issues) or outputCommitter.setupJob() throws (output directory exists/inaccessible).","commonSituations":"Wrong mapreduce.outputformat.class configured; HDFS output directory already exists; invalid job configuration (e.g. missing output path); incompatible Hadoop version's OutputFormat behavior; transient NameNode connectivity failure.","solutions":["Verify the OutputFormat class in configuration is correct, instantiable, and matches your sink (createOutputFormatFromConfig).","Ensure the Hadoop Job configuration (output dir, filesystem, Kerberos creds) is valid before running the pipeline.","Delete/ensure the output directory does not exist or the committer is allowed to set it up.","Inspect the wrapped cause 'e' in the exception for the real Hadoop error."],"exampleFix":"// before\nconf.set(\"mapreduce.outputformat.class\", \"org.apache.hadoop.mapreduce.lib.output.TextOutputFormat\");\n// after (with explicit output dir set up)\nconf.set(\"mapreduce.outputformat.class\", \"org.apache.hadoop.mapreduce.lib.output.TextOutputFormat\");\nFileOutputFormat.setOutputPath(conf, new Path(\"/out/\" + UUID.randomUUID()));","handlingStrategy":"try-catch","validationCode":"// validate before running sink\nString fmtClass = conf.get(\"mapreduce.outputformat.class\");\nClass<?> cls = Class.forName(fmtClass);\nif (cls.isInterface() || Modifier.isAbstract(cls.getModifiers())) throw new IllegalArgumentException(fmtClass + \" not instantiable\");\ncls.getConstructor();","typeGuard":null,"tryCatchPattern":"try {\n  pipeline.run().waitUntilFinish();\n} catch (IllegalStateException e) {\n  if (e.getMessage().contains(\"Unable to create OutputCommitter\")) {\n    Throwable cause = e.getCause();\n    LOG.error(\"OutputCommitter setup failed: {}\", cause, cause);\n  }\n  throw e;\n}","preventionTips":["Always set mapreduce.outputformat.class to a concrete class with a public no-arg constructor.","Pre-create or clear the output directory and verify write permissions before launching.","Run a small test job against the same Hadoop configuration before full pipeline runs.","Log and inspect the cause chain — the real Hadoop error is always wrapped."],"tags":["hadoop","output-committer","hadoopformatio","job-setup"],"backgroundTag":"invalid-config-value","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}