apache/hadoop · error · IOException

DistCp failed. jobId={jobId} failure={failureInfo}

Error message

DistCp failed. jobId={jobId} failure={failureInfo}

What it means

During the DIFF_DISTCP stage, DistCpProcedure polls the MapReduce distcp job that copies the diff between the LAST and CURRENT snapshots on src. When that job is complete but not successful, it throws IOException carrying the jobId and job.getFailureInfo(). The procedure framework persists state in its journal, so after fixing the root cause you re-run fedbalance and it resumes rather than copying everything again.

Source

Thrown at hadoop-tools/hadoop-federation-balance/src/main/java/org/apache/hadoop/tools/fedbalance/DistCpProcedure.java:235

      jobId = submitDistCpJob(
          src.toString() + HdfsConstants.SEPARATOR_DOT_SNAPSHOT_DIR_SEPARATOR
              + CURRENT_SNAPSHOT_NAME, dst.toString(), false);
    }
  }

  /**
   * The distcp copying diffs between LAST_SNAPSHOT_NAME and
   * CURRENT_SNAPSHOT_NAME.
   */
  void diffDistCp() throws IOException, RetryException {
    RunningJobStatus job = getCurrentJob();
    if (job != null) {
      if (job.isComplete()) {
        jobId = null;
        if (job.isSuccessful()) {
          LOG.info("DistCp succeeded. jobId={}", job.getJobID());
        } else {
          throw new IOException("DistCp failed. jobId=" + job.getJobID()
              + " failure=" + job.getFailureInfo());
        }
      } else {
        throw new RetryException(); // wait job complete.
      }
    } else if (diffDistCpStageDone()) {
      updateStage(Stage.DISABLE_WRITE);
    } else {
      submitDiffDistCp();
    }
  }

  /**
   * Disable write by cancelling the execute permission of the source path.
   * TODO: Disable the super user from writing.
   * @param fbcontext the context.
   * @throws IOException if can't disable write.
   */

View on GitHub (pinned to 2add963021)

Solutions

1. Take the printed jobId, find the YARN application ('yarn logs -applicationId <appId>' or JobHistory UI) and fix the underlying failure.
2. Re-run the fedbalance command afterwards; the procedure resumes from its journal and only redoes the diff stage.
3. Check dst-side permissions, space and quotas, and that fedbalance's snapshots on src still exist.
4. If the diff keeps failing because src churns, run with a larger diff threshold or force-close open files earlier so the final diff is smaller.

try {
  job = FedBalance.submit(opts);   // procedure runs DIFF_DISTCP internally
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().startsWith("DistCp failed.")) {
    // jobId is embedded in the message; inspect YARN/JobHistory, fix cause, re-run to resume
  } else {
    throw e;
  }
}

When it happens

Trigger: In diffDistCp(): getCurrentJob() returns a job with isComplete()==true and isSuccessful()==false. That is, the diff distcp MR job failed at runtime - map tasks died, commit failed, tasks killed by RM/NM.

Common situations: Datanode or Namenode connectivity problems during the diff copy; dst permission or quota errors at commit time; checksum mismatches on read; cluster contention causing task preemption.

Related errors

• Final DistCp failed. Failure: {failureInfo}
• {src} should be a directory.
• {dst} already exists.
• {src} shouldn't enable snapshot.
• The dst path={dst} already exists. The admin should delete i

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/20a4bc1121b63ab1. Report an issue: GitHub.