{"record":{"id":"20a4bc1121b63ab1","repo":"apache/hadoop","slug":"distcp-failed-jobid-jobid-failure-failureinfo","errorCode":null,"errorMessage":"DistCp failed. jobId={jobId} failure={failureInfo}","messagePattern":"DistCp failed\\. jobId=(.+?) failure=(.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-tools/hadoop-federation-balance/src/main/java/org/apache/hadoop/tools/fedbalance/DistCpProcedure.java","lineNumber":235,"sourceCode":"      jobId = submitDistCpJob(\n          src.toString() + HdfsConstants.SEPARATOR_DOT_SNAPSHOT_DIR_SEPARATOR\n              + CURRENT_SNAPSHOT_NAME, dst.toString(), false);\n    }\n  }\n\n  /**\n   * The distcp copying diffs between LAST_SNAPSHOT_NAME and\n   * CURRENT_SNAPSHOT_NAME.\n   */\n  void diffDistCp() throws IOException, RetryException {\n    RunningJobStatus job = getCurrentJob();\n    if (job != null) {\n      if (job.isComplete()) {\n        jobId = null;\n        if (job.isSuccessful()) {\n          LOG.info(\"DistCp succeeded. jobId={}\", job.getJobID());\n        } else {\n          throw new IOException(\"DistCp failed. jobId=\" + job.getJobID()\n              + \" failure=\" + job.getFailureInfo());\n        }\n      } else {\n        throw new RetryException(); // wait job complete.\n      }\n    } else if (diffDistCpStageDone()) {\n      updateStage(Stage.DISABLE_WRITE);\n    } else {\n      submitDiffDistCp();\n    }\n  }\n\n  /**\n   * Disable write by cancelling the execute permission of the source path.\n   * TODO: Disable the super user from writing.\n   * @param fbcontext the context.\n   * @throws IOException if can't disable write.\n   */","sourceCodeStart":217,"sourceCodeEnd":253,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-tools/hadoop-federation-balance/src/main/java/org/apache/hadoop/tools/fedbalance/DistCpProcedure.java#L217-L253","documentation":"During the DIFF_DISTCP stage, DistCpProcedure polls the MapReduce distcp job that copies the diff between the LAST and CURRENT snapshots on src. When that job is complete but not successful, it throws IOException carrying the jobId and job.getFailureInfo(). The procedure framework persists state in its journal, so after fixing the root cause you re-run fedbalance and it resumes rather than copying everything again.","triggerScenarios":"In diffDistCp(): getCurrentJob() returns a job with isComplete()==true and isSuccessful()==false. That is, the diff distcp MR job failed at runtime - map tasks died, commit failed, tasks killed by RM/NM.","commonSituations":"Datanode or Namenode connectivity problems during the diff copy; dst permission or quota errors at commit time; checksum mismatches on read; cluster contention causing task preemption.","solutions":["Take the printed jobId, find the YARN application ('yarn logs -applicationId <appId>' or JobHistory UI) and fix the underlying failure.","Re-run the fedbalance command afterwards; the procedure resumes from its journal and only redoes the diff stage.","Check dst-side permissions, space and quotas, and that fedbalance's snapshots on src still exist.","If the diff keeps failing because src churns, run with a larger diff threshold or force-close open files earlier so the final diff is smaller."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  job = FedBalance.submit(opts);   // procedure runs DIFF_DISTCP internally\n} catch (IOException e) {\n  if (e.getMessage() != null && e.getMessage().startsWith(\"DistCp failed.\")) {\n    // jobId is embedded in the message; inspect YARN/JobHistory, fix cause, re-run to resume\n  } else {\n    throw e;\n  }\n}","preventionTips":["Monitor the diff distcp MR job while fedbalance runs so failures are caught early.","Keep the journal (hdfs.fedbalance.procedure.scheduler.journal.uri) intact so re-runs resume instead of restarting.","Pre-verify dst permissions, space and quotas before balancing."],"tags":["fedbalance","federation-balance","distcp","mapreduce","diff-copy"],"backgroundTag":"mapreduce-job-failed","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}