{"record":{"id":"d190dac5c3a7b776","repo":"apache/hadoop","slug":"working-directory-not-configured","errorCode":null,"errorMessage":"Working directory not configured","messagePattern":"Working directory not configured","errorType":"exception","errorClass":"IllegalStateException","httpStatus":null,"severity":"error","filePath":"hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyOutputFormat.java","lineNumber":120,"sourceCode":"\n  /** {@inheritDoc} */\n  @Override\n  public OutputCommitter getOutputCommitter(TaskAttemptContext context) throws IOException {\n    return new CopyCommitter(getOutputPath(context), context);\n  }\n\n  /** {@inheritDoc} */\n  @Override\n  public void checkOutputSpecs(JobContext context) throws IOException {\n    Configuration conf = context.getConfiguration();\n\n    if (getCommitDirectory(conf) == null) {\n      throw new IllegalStateException(\"Commit directory not configured\");\n    }\n\n    Path workingPath = getWorkingDirectory(conf);\n    if (workingPath == null) {\n      throw new IllegalStateException(\"Working directory not configured\");\n    }\n\n    // get delegation token for outDir's file system\n    TokenCache.obtainTokensForNamenodes(context.getCredentials(),\n                                        new Path[] {workingPath}, conf);\n  }\n}\n","sourceCodeStart":102,"sourceCodeEnd":128,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyOutputFormat.java#L102-L128","documentation":"Companion check in CopyOutputFormat.checkOutputSpecs(): distcp.target.work.path (set via CopyOutputFormat.setWorkingDirectory(); with -atomic it is the temp dir, otherwise it equals the final path) must be set or the job is rejected at submission. Hitting it means the job was assembled without the required distcp work-path configuration.","triggerScenarios":"Building a Job with CopyOutputFormat directly and never calling setWorkingDirectory(job, path); rebuilding a Configuration and copying only some distcp.* properties; -atomic setups where the work-path key was dropped.","commonSituations":"Custom embedding of distcp's output format; pipeline code that constructs a fresh Configuration and manually transfers settings; job XML round-trips losing keys.","solutions":["Call CopyOutputFormat.setWorkingDirectory(job, path) before submit","Or set \"distcp.target.work.path\" directly in the job Configuration (for -atomic, point it at a temp dir distinct from the final path)","Use the DistCp driver/API instead of hand-assembling the MapReduce job","Log the distcp.* keys before submit to confirm both work and final paths are set"],"exampleFix":"// before\njob.setOutputFormatClass(CopyOutputFormat.class);\nCopyOutputFormat.setCommitDirectory(job, finalPath);\njob.submit();               // IllegalStateException: work dir unset\n\n// after\njob.setOutputFormatClass(CopyOutputFormat.class);\nCopyOutputFormat.setWorkingDirectory(job, useAtomic ? tempWorkPath : finalPath);\nCopyOutputFormat.setCommitDirectory(job, finalPath);\njob.submit();","handlingStrategy":"validation","validationCode":"// before job.submit()\nif (conf.get(DistCpConstants.CONF_LABEL_TARGET_WORK_PATH) == null) {\n  CopyOutputFormat.setWorkingDirectory(job, useAtomic ? tempWorkPath : targetPath);\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Always pair setWorkingDirectory with setCommitDirectory when embedding CopyOutputFormat","For -atomic, point the work path at a temp dir distinct from the final path","Log the distcp.* configuration keys before submission in CI for embedded jobs"],"tags":["distcp","hadoop","mapreduce","configuration","job-setup"],"backgroundTag":"missing-configuration","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}