{"record":{"id":"80ab75974e43c96d","repo":"apache/hadoop","slug":"commit-directory-not-configured","errorCode":null,"errorMessage":"Commit directory not configured","messagePattern":"Commit directory not configured","errorType":"exception","errorClass":"IllegalStateException","httpStatus":null,"severity":"error","filePath":"hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyOutputFormat.java","lineNumber":115,"sourceCode":"      return null;\n    } else {\n      return new Path(commitDirectory);\n    }\n  }\n\n  /** {@inheritDoc} */\n  @Override\n  public OutputCommitter getOutputCommitter(TaskAttemptContext context) throws IOException {\n    return new CopyCommitter(getOutputPath(context), context);\n  }\n\n  /** {@inheritDoc} */\n  @Override\n  public void checkOutputSpecs(JobContext context) throws IOException {\n    Configuration conf = context.getConfiguration();\n\n    if (getCommitDirectory(conf) == null) {\n      throw new IllegalStateException(\"Commit directory not configured\");\n    }\n\n    Path workingPath = getWorkingDirectory(conf);\n    if (workingPath == null) {\n      throw new IllegalStateException(\"Working directory not configured\");\n    }\n\n    // get delegation token for outDir's file system\n    TokenCache.obtainTokensForNamenodes(context.getCredentials(),\n                                        new Path[] {workingPath}, conf);\n  }\n}\n","sourceCodeStart":97,"sourceCodeEnd":128,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyOutputFormat.java#L97-L128","documentation":"CopyOutputFormat.checkOutputSpecs() runs at job submission and requires distcp.target.final.path (normally set via CopyOutputFormat.setCommitDirectory() or automatically by the DistCp driver). A null value means the MapReduce job uses distcp's output format without the distcp commit-directory setup, and the job is rejected before it starts.","triggerScenarios":"Programmatically building a Job with CopyOutputFormat as the output format without calling CopyOutputFormat.setCommitDirectory(job, path); a Configuration that was copied or sanitized and lost its distcp.* keys; reusing a stale JobConf that never had the key.","commonSituations":"Custom tools embedding CopyMapper/CopyOutputFormat directly; config overlays that blank out distcp.target.final.path; moving a hand-built job to another cluster.","solutions":["Call CopyOutputFormat.setCommitDirectory(job, new Path(\"hdfs://nn/dst\")) before submit","Or set the key directly: job.getConfiguration().set(\"distcp.target.final.path\", \"hdfs://nn/dst\")","Prefer driving distcp through the DistCp API (DistCpOptions + DistCp) or the CLI, which set this automatically","Dump the job configuration before submit and verify the key is present"],"exampleFix":"// before\nJob job = Job.getInstance(conf);\njob.setOutputFormatClass(CopyOutputFormat.class);\njob.submit();                     // IllegalStateException: commit dir unset\n\n// after\nJob job = Job.getInstance(conf);\njob.setOutputFormatClass(CopyOutputFormat.class);\nCopyOutputFormat.setCommitDirectory(job, new Path(\"hdfs://nn/dst\"));\njob.submit();","handlingStrategy":"validation","validationCode":"// before job.submit()\nif (conf.get(DistCpConstants.CONF_LABEL_TARGET_FINAL_PATH) == null) {\n  CopyOutputFormat.setCommitDirectory(job, targetPath);\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Prefer the DistCp API/CLI over hand-assembling the MapReduce job","When reusing Configurations, copy the whole distcp.* key set","Assert both distcp.target.work.path and distcp.target.final.path before submit"],"tags":["distcp","hadoop","mapreduce","configuration","job-setup"],"backgroundTag":"missing-configuration","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}