apache/hadoop · critical · IOException

DistCp failure: Job " + job.getJobID() + " has failed: " + j

Error message

DistCp failure: Job " + job.getJobID() + " has failed: " + job.getStatus().getFailureInfo()

What it means

DistCp.waitForJobCompletion blocks on job.waitForCompletion(true); when the MapReduce job does not finish successfully it throws IOException('DistCp failure: Job <id> has failed: <failureInfo>'). The failureInfo string comes from the job status and usually carries the first task-side exception, so this message is the pointer to the real root cause, not the cause itself.

Source

Thrown at hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java:257

    String jobID = job.getJobID().toString();
    job.getConfiguration().set(DistCpConstants.CONF_LABEL_DISTCP_JOB_ID,
        jobID);
    // Set the jobId for the applications running through run method.
    getConf().set(DistCpConstants.CONF_LABEL_DISTCP_JOB_ID, jobID);
    LOG.info("DistCp job-id: " + jobID);

    return job;
  }

  /**
   * Wait for the given job to complete.
   * @param job the given mapreduce job that has already been submitted
   */
  public void waitForJobCompletion(Job job) throws Exception {
    assert job != null;
    if (!job.waitForCompletion(true)) {
      throw new IOException("DistCp failure: Job " + job.getJobID()
          + " has failed: " + job.getStatus().getFailureInfo());
    }
  }

  /**
   * Set targetPathExists in both inputOptions and job config,
   * for the benefit of CopyCommitter
   */
  private void setTargetPathExists() throws IOException {
    Path target = context.getTargetPath();
    FileSystem targetFS = target.getFileSystem(getConf());
    boolean targetExists = targetFS.exists(target);
    context.setTargetPathExists(targetExists);
    getConf().setBoolean(DistCpConstants.CONF_LABEL_TARGET_PATH_EXISTS, 
        targetExists);
  }

  /**

View on GitHub (pinned to 2add963021)

Solutions

Read the failureInfo embedded after 'has failed:', then pull task logs with yarn logs -applicationId <appId> (the distcp job id maps to the YARN application)
Fix the underlying cause: permissions, missing sources, connectivity, or raise task memory (-Dmapreduce.map.memory.mb) if tasks were killed
Re-run with -update so already-copied files are skipped instead of recopying everything
Use -atomic (with a same-FS tmp dir) on future runs so failed jobs leave no partial output at the target

Example fix

# before
hadoop distcp hdfs://src hdfs://dst
# -> DistCp failure: Job job_1690000000000_0042 has failed: Too many fetch failures ...

# after: inspect logs, then resume incrementally
yarn logs -applicationId application_1690000000000_0042 > job.log
hadoop distcp -update hdfs://src hdfs://dst

Defensive patterns

Strategy: retry

Try / catch

try {
  new DistCp(conf, options).execute();
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().contains("has failed:")) {
    // failureInfo follows 'has failed:'; pull yarn logs for the job's application,
    // fix the reported root cause, then resume with -update to skip copied files
    LOG.warn("DistCp job failed; after fixing the cause re-run with -update");
  } else {
    throw e;
  }
}

Prevention

Pre-check source readability and target writability before submitting the job
Use -update for large/incremental copies so a retry is cheap
Capture yarn logs promptly after failure before log aggregation ages them out
Size -m and map memory to the workload to avoid task OOM kills

When it happens

Trigger: Any failed or killed map task during the copy: source file deleted after listing, permission denied on target, NameNode/DataNode or object-store connectivity errors, checksum mismatches, task memory limits, or the job being killed by an operator or YARN preemption.

Common situations: Large copies hitting transient cluster issues; incorrect permissions/ownership on the destination; object-store targets throttling (HTTP 503 from S3A/ABFS); source data modified mid-copy; job killed for queue/preemption reasons.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/105c720f7632c85c. Report an issue: GitHub.