apache/hadoop · error · IllegalStateException

Commit directory not configured

Error message

Commit directory not configured

What it means

CopyOutputFormat.checkOutputSpecs() runs at job submission and requires distcp.target.final.path (normally set via CopyOutputFormat.setCommitDirectory() or automatically by the DistCp driver). A null value means the MapReduce job uses distcp's output format without the distcp commit-directory setup, and the job is rejected before it starts.

Source

Thrown at hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyOutputFormat.java:115

      return null;
    } else {
      return new Path(commitDirectory);
    }
  }

  /** {@inheritDoc} */
  @Override
  public OutputCommitter getOutputCommitter(TaskAttemptContext context) throws IOException {
    return new CopyCommitter(getOutputPath(context), context);
  }

  /** {@inheritDoc} */
  @Override
  public void checkOutputSpecs(JobContext context) throws IOException {
    Configuration conf = context.getConfiguration();

    if (getCommitDirectory(conf) == null) {
      throw new IllegalStateException("Commit directory not configured");
    }

    Path workingPath = getWorkingDirectory(conf);
    if (workingPath == null) {
      throw new IllegalStateException("Working directory not configured");
    }

    // get delegation token for outDir's file system
    TokenCache.obtainTokensForNamenodes(context.getCredentials(),
                                        new Path[] {workingPath}, conf);
  }
}

View on GitHub (pinned to 2add963021)

Solutions

1. Call CopyOutputFormat.setCommitDirectory(job, new Path("hdfs://nn/dst")) before submit
2. Or set the key directly: job.getConfiguration().set("distcp.target.final.path", "hdfs://nn/dst")
3. Prefer driving distcp through the DistCp API (DistCpOptions + DistCp) or the CLI, which set this automatically
4. Dump the job configuration before submit and verify the key is present

Example fix

// before
Job job = Job.getInstance(conf);
job.setOutputFormatClass(CopyOutputFormat.class);
job.submit();                     // IllegalStateException: commit dir unset

// after
Job job = Job.getInstance(conf);
job.setOutputFormatClass(CopyOutputFormat.class);
CopyOutputFormat.setCommitDirectory(job, new Path("hdfs://nn/dst"));
job.submit();

// before job.submit()
if (conf.get(DistCpConstants.CONF_LABEL_TARGET_FINAL_PATH) == null) {
  CopyOutputFormat.setCommitDirectory(job, targetPath);
}

When it happens

Trigger: Programmatically building a Job with CopyOutputFormat as the output format without calling CopyOutputFormat.setCommitDirectory(job, path); a Configuration that was copied or sanitized and lost its distcp.* keys; reusing a stale JobConf that never had the key.

Common situations: Custom tools embedding CopyMapper/CopyOutputFormat directly; config overlays that blank out distcp.target.final.path; moving a hand-built job to another cluster.

Related errors

• Working directory not configured
• Invalid specification for distributed-cache artifacts of typ
• Unable to parse '{}' as a URI, check the setting for mapredu
• Could not locate MapReduce framework name '{}' in mapreduce.
• MapReduce JobHistory WebApp Address does not contain a valid

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/80ab75974e43c96d. Report an issue: GitHub.