apache/hadoop · error · IllegalStateException

Working directory not configured

Error message

Working directory not configured

What it means

Companion check in CopyOutputFormat.checkOutputSpecs(): distcp.target.work.path (set via CopyOutputFormat.setWorkingDirectory(); with -atomic it is the temp dir, otherwise it equals the final path) must be set or the job is rejected at submission. Hitting it means the job was assembled without the required distcp work-path configuration.

Source

Thrown at hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyOutputFormat.java:120

  /** {@inheritDoc} */
  @Override
  public OutputCommitter getOutputCommitter(TaskAttemptContext context) throws IOException {
    return new CopyCommitter(getOutputPath(context), context);
  }

  /** {@inheritDoc} */
  @Override
  public void checkOutputSpecs(JobContext context) throws IOException {
    Configuration conf = context.getConfiguration();

    if (getCommitDirectory(conf) == null) {
      throw new IllegalStateException("Commit directory not configured");
    }

    Path workingPath = getWorkingDirectory(conf);
    if (workingPath == null) {
      throw new IllegalStateException("Working directory not configured");
    }

    // get delegation token for outDir's file system
    TokenCache.obtainTokensForNamenodes(context.getCredentials(),
                                        new Path[] {workingPath}, conf);
  }
}

View on GitHub (pinned to 2add963021)

Solutions

1. Call CopyOutputFormat.setWorkingDirectory(job, path) before submit
2. Or set "distcp.target.work.path" directly in the job Configuration (for -atomic, point it at a temp dir distinct from the final path)
3. Use the DistCp driver/API instead of hand-assembling the MapReduce job
4. Log the distcp.* keys before submit to confirm both work and final paths are set

Example fix

// before
job.setOutputFormatClass(CopyOutputFormat.class);
CopyOutputFormat.setCommitDirectory(job, finalPath);
job.submit();               // IllegalStateException: work dir unset

// after
job.setOutputFormatClass(CopyOutputFormat.class);
CopyOutputFormat.setWorkingDirectory(job, useAtomic ? tempWorkPath : finalPath);
CopyOutputFormat.setCommitDirectory(job, finalPath);
job.submit();

// before job.submit()
if (conf.get(DistCpConstants.CONF_LABEL_TARGET_WORK_PATH) == null) {
  CopyOutputFormat.setWorkingDirectory(job, useAtomic ? tempWorkPath : targetPath);
}

When it happens

Trigger: Building a Job with CopyOutputFormat directly and never calling setWorkingDirectory(job, path); rebuilding a Configuration and copying only some distcp.* properties; -atomic setups where the work-path key was dropped.

Common situations: Custom embedding of distcp's output format; pipeline code that constructs a fresh Configuration and manually transfers settings; job XML round-trips losing keys.

Related errors

• Commit directory not configured
• Invalid specification for distributed-cache artifacts of typ
• Unable to parse '{}' as a URI, check the setting for mapredu
• Could not locate MapReduce framework name '{}' in mapreduce.
• MapReduce JobHistory WebApp Address does not contain a valid

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/d190dac5c3a7b776. Report an issue: GitHub.