{"record":{"id":"105c720f7632c85c","repo":"apache/hadoop","slug":"distcp-failure-job-job-getjobid-has-fai","errorCode":null,"errorMessage":"DistCp failure: Job \" + job.getJobID() + \" has failed: \" + job.getStatus().getFailureInfo()","messagePattern":"DistCp failure: Job \" \\+ job\\.getJobID\\(\\) \\+ \" has failed: \" \\+ job\\.getStatus\\(\\)\\.getFailureInfo\\(\\)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"critical","filePath":"hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java","lineNumber":257,"sourceCode":"\n    String jobID = job.getJobID().toString();\n    job.getConfiguration().set(DistCpConstants.CONF_LABEL_DISTCP_JOB_ID,\n        jobID);\n    // Set the jobId for the applications running through run method.\n    getConf().set(DistCpConstants.CONF_LABEL_DISTCP_JOB_ID, jobID);\n    LOG.info(\"DistCp job-id: \" + jobID);\n\n    return job;\n  }\n\n  /**\n   * Wait for the given job to complete.\n   * @param job the given mapreduce job that has already been submitted\n   */\n  public void waitForJobCompletion(Job job) throws Exception {\n    assert job != null;\n    if (!job.waitForCompletion(true)) {\n      throw new IOException(\"DistCp failure: Job \" + job.getJobID()\n          + \" has failed: \" + job.getStatus().getFailureInfo());\n    }\n  }\n\n  /**\n   * Set targetPathExists in both inputOptions and job config,\n   * for the benefit of CopyCommitter\n   */\n  private void setTargetPathExists() throws IOException {\n    Path target = context.getTargetPath();\n    FileSystem targetFS = target.getFileSystem(getConf());\n    boolean targetExists = targetFS.exists(target);\n    context.setTargetPathExists(targetExists);\n    getConf().setBoolean(DistCpConstants.CONF_LABEL_TARGET_PATH_EXISTS, \n        targetExists);\n  }\n\n  /**","sourceCodeStart":239,"sourceCodeEnd":275,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java#L239-L275","documentation":"DistCp.waitForJobCompletion blocks on job.waitForCompletion(true); when the MapReduce job does not finish successfully it throws IOException('DistCp failure: Job <id> has failed: <failureInfo>'). The failureInfo string comes from the job status and usually carries the first task-side exception, so this message is the pointer to the real root cause, not the cause itself.","triggerScenarios":"Any failed or killed map task during the copy: source file deleted after listing, permission denied on target, NameNode/DataNode or object-store connectivity errors, checksum mismatches, task memory limits, or the job being killed by an operator or YARN preemption.","commonSituations":"Large copies hitting transient cluster issues; incorrect permissions/ownership on the destination; object-store targets throttling (HTTP 503 from S3A/ABFS); source data modified mid-copy; job killed for queue/preemption reasons.","solutions":["Read the failureInfo embedded after 'has failed:', then pull task logs with yarn logs -applicationId <appId> (the distcp job id maps to the YARN application)","Fix the underlying cause: permissions, missing sources, connectivity, or raise task memory (-Dmapreduce.map.memory.mb) if tasks were killed","Re-run with -update so already-copied files are skipped instead of recopying everything","Use -atomic (with a same-FS tmp dir) on future runs so failed jobs leave no partial output at the target"],"exampleFix":"# before\nhadoop distcp hdfs://src hdfs://dst\n# -> DistCp failure: Job job_1690000000000_0042 has failed: Too many fetch failures ...\n\n# after: inspect logs, then resume incrementally\nyarn logs -applicationId application_1690000000000_0042 > job.log\nhadoop distcp -update hdfs://src hdfs://dst","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  new DistCp(conf, options).execute();\n} catch (IOException e) {\n  if (e.getMessage() != null && e.getMessage().contains(\"has failed:\")) {\n    // failureInfo follows 'has failed:'; pull yarn logs for the job's application,\n    // fix the reported root cause, then resume with -update to skip copied files\n    LOG.warn(\"DistCp job failed; after fixing the cause re-run with -update\");\n  } else {\n    throw e;\n  }\n}","preventionTips":["Pre-check source readability and target writability before submitting the job","Use -update for large/incremental copies so a retry is cheap","Capture yarn logs promptly after failure before log aggregation ages them out","Size -m and map memory to the workload to avoid task OOM kills"],"tags":["distcp","mapreduce","job-failure","yarn"],"backgroundTag":"mapreduce-job-failure","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}