apache/hadoop · error · IOException

Final DistCp failed. Failure: {failureInfo}

Error message

Final DistCp failed. Failure: {failureInfo}

What it means

finalDistCp() runs after closeAllOpenFiles(srcFs, src): it polls the final diff-distcp job and, when the job completed unsuccessfully, throws IOException with job.getFailureInfo(). At this point writes on src were disabled and open files force-closed, so src stays read-only until the procedure finishes (finish() calls enableWrite()). The copy state is journaled, so a re-run resubmits only the final diff.

Source

Thrown at hadoop-tools/hadoop-federation-balance/src/main/java/org/apache/hadoop/tools/fedbalance/DistCpProcedure.java:301

  }

  /**
   * Close all open files then submit the distcp with -diff.
   */
  void finalDistCp() throws IOException, RetryException {
    // Close all open files then do the final distcp.
    closeAllOpenFiles(srcFs, src);
    // Final distcp.
    RunningJobStatus job = getCurrentJob();
    if (job != null) {
      // the distcp has been submitted.
      if (job.isComplete()) {
        jobId = null; // unset jobId because the job is done.
        if (job.isSuccessful()) {
          updateStage(Stage.FINISH);
          return;
        } else {
          throw new IOException(
              "Final DistCp failed. Failure: " + job.getFailureInfo());
        }
      } else {
        throw new RetryException();
      }
    } else {
      submitDiffDistCp();
    }
  }

  void finish() throws IOException {
    enableWrite();
    if (srcFs.exists(src)) {
      cleanupSnapshot(srcFs, src);
    }
    if (dstFs.exists(dst)) {
      cleanupSnapshot(dstFs, dst);
    }

View on GitHub (pinned to 2add963021)

Solutions

1. Read the failure info string embedded in the message and inspect that MR job's logs for the root cause.
2. Fix the cause, then re-run the fedbalance job: it re-enters FINAL_DISTCP and resubmits only the final diff distcp.
3. If you must abandon the procedure, manually re-enable writes on src, and clean up dst and fedbalance snapshots before the next attempt.
4. Verify dst still has space/quota for the final diff before re-running.

try {
  // run balance procedure through FINAL_DISTCP
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().startsWith("Final DistCp failed.")) {
    // read job.getFailureInfo() from the message, fix the MR failure, re-run fedbalance to resume
    // remember src is write-disabled and open files were force-closed at this stage
  }
}

When it happens

Trigger: In the FINAL_DISTCP stage: getCurrentJob() yields a job with isComplete()==true and isSuccessful()==false after open files were already closed on src.

Common situations: RM/NM restarts killing the final MR job; dst quota or permission problems at commit; network partition between clusters during the last incremental copy.

Related errors

• DistCp failed. jobId={jobId} failure={failureInfo}
• {src} should be a directory.
• {dst} already exists.
• {src} shouldn't enable snapshot.
• The dst path={dst} already exists. The admin should delete i

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/0abee572160182cf. Report an issue: GitHub.