{"record":{"id":"0abee572160182cf","repo":"apache/hadoop","slug":"final-distcp-failed-failure-failureinfo","errorCode":null,"errorMessage":"Final DistCp failed. Failure: {failureInfo}","messagePattern":"Final DistCp failed\\. Failure: (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-tools/hadoop-federation-balance/src/main/java/org/apache/hadoop/tools/fedbalance/DistCpProcedure.java","lineNumber":301,"sourceCode":"  }\n\n  /**\n   * Close all open files then submit the distcp with -diff.\n   */\n  void finalDistCp() throws IOException, RetryException {\n    // Close all open files then do the final distcp.\n    closeAllOpenFiles(srcFs, src);\n    // Final distcp.\n    RunningJobStatus job = getCurrentJob();\n    if (job != null) {\n      // the distcp has been submitted.\n      if (job.isComplete()) {\n        jobId = null; // unset jobId because the job is done.\n        if (job.isSuccessful()) {\n          updateStage(Stage.FINISH);\n          return;\n        } else {\n          throw new IOException(\n              \"Final DistCp failed. Failure: \" + job.getFailureInfo());\n        }\n      } else {\n        throw new RetryException();\n      }\n    } else {\n      submitDiffDistCp();\n    }\n  }\n\n  void finish() throws IOException {\n    enableWrite();\n    if (srcFs.exists(src)) {\n      cleanupSnapshot(srcFs, src);\n    }\n    if (dstFs.exists(dst)) {\n      cleanupSnapshot(dstFs, dst);\n    }","sourceCodeStart":283,"sourceCodeEnd":319,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-tools/hadoop-federation-balance/src/main/java/org/apache/hadoop/tools/fedbalance/DistCpProcedure.java#L283-L319","documentation":"finalDistCp() runs after closeAllOpenFiles(srcFs, src): it polls the final diff-distcp job and, when the job completed unsuccessfully, throws IOException with job.getFailureInfo(). At this point writes on src were disabled and open files force-closed, so src stays read-only until the procedure finishes (finish() calls enableWrite()). The copy state is journaled, so a re-run resubmits only the final diff.","triggerScenarios":"In the FINAL_DISTCP stage: getCurrentJob() yields a job with isComplete()==true and isSuccessful()==false after open files were already closed on src.","commonSituations":"RM/NM restarts killing the final MR job; dst quota or permission problems at commit; network partition between clusters during the last incremental copy.","solutions":["Read the failure info string embedded in the message and inspect that MR job's logs for the root cause.","Fix the cause, then re-run the fedbalance job: it re-enters FINAL_DISTCP and resubmits only the final diff distcp.","If you must abandon the procedure, manually re-enable writes on src, and clean up dst and fedbalance snapshots before the next attempt.","Verify dst still has space/quota for the final diff before re-running."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  // run balance procedure through FINAL_DISTCP\n} catch (IOException e) {\n  if (e.getMessage() != null && e.getMessage().startsWith(\"Final DistCp failed.\")) {\n    // read job.getFailureInfo() from the message, fix the MR failure, re-run fedbalance to resume\n    // remember src is write-disabled and open files were force-closed at this stage\n  }\n}","preventionTips":["Confirm dst quota/space before the final diff copy, not just at submit time.","Avoid restarting YARN nodes mid-balance; the final stage is small but fatal to lose.","If abandoning at this stage, plan manual cleanup: re-enable writes on src, remove dst and fedbalance snapshots."],"tags":["fedbalance","federation-balance","distcp","mapreduce","final-copy"],"backgroundTag":"mapreduce-job-failed","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}