{"record":{"id":"a4b44f458ced8dde","repo":"apache/hadoop","slug":"submit-job-failed","errorCode":null,"errorMessage":"Submit job failed.","messagePattern":"Submit job failed\\.","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-tools/hadoop-federation-balance/src/main/java/org/apache/hadoop/tools/fedbalance/DistCpProcedure.java","lineNumber":504,"sourceCode":"    command.add(mapNum + \"\");\n    command.add(\"-bandwidth\");\n    command.add(bandWidth + \"\");\n    command.add(srcParam);\n    command.add(dstParam);\n\n    Configuration config = new Configuration(conf);\n    DistCp distCp;\n    try {\n      distCp = new DistCp(config,\n          OptionsParser.parse(command.toArray(new String[]{})));\n      Job job = distCp.createAndSubmitJob();\n      LOG.info(\"Submit distcp job={}\", job);\n      if (enabledForTest) {\n        localJob = job;\n      }\n      return job.getJobID().toString();\n    } catch (Exception e) {\n      throw new IOException(\"Submit job failed.\", e);\n    }\n  }\n\n  @Override\n  public void write(DataOutput out) throws IOException {\n    super.write(out);\n    context.write(out);\n    if (jobId == null) {\n      out.writeBoolean(false);\n    } else {\n      out.writeBoolean(true);\n      Text.writeString(out, jobId);\n    }\n    out.writeInt(stage.ordinal());\n    if (fPerm == null) {\n      out.writeBoolean(false);\n    } else {\n      out.writeBoolean(true);","sourceCodeStart":486,"sourceCodeEnd":522,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-tools/hadoop-federation-balance/src/main/java/org/apache/hadoop/tools/fedbalance/DistCpProcedure.java#L486-L522","documentation":"submitDistCpJob() wraps the entire distcp submission - OptionsParser.parse() of the assembled command, constructing the DistCp object, and createAndSubmitJob() - in a single IOException('Submit job failed.', e). The meaningful diagnostic is always the chained cause; the wrapper message alone carries no information. It fires before any mapper runs, so no data has been copied when you see it.","triggerScenarios":"Any exception from: (a) invalid distcp options assembled from fedbalance parameters (bad bandwidth or map count), (b) DistCp input validation rejecting src/dst options, or (c) MapReduce submission failing - ResourceManager unreachable, staging directory issues, missing delegation tokens in cross-cluster federation setups.","commonSituations":"Mistyped -bandwidth/-map values passed through FedBalanceOptions; YARN not running or rm address misconfigured; fs.defaultFS or mapreduce.jobtracker.staging.dir problems; router-based federation without tokens for the remote nameservice.","solutions":["Read the cause chain first: find 'Caused by:' under this exception in the log and fix that specific error.","Verify MR submission works at all: run a trivial 'hadoop distcp' or 'hadoop jar hadoop-mapreduce-examples' job on the same client.","Check the fedbalance parameters (map count, bandwidth) are within distcp's accepted ranges.","For cross-cluster runs, confirm delegation tokens / keytab for both nameservices are available to the client."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  FedBalance.main(args);\n} catch (IOException e) {\n  if (\"Submit job failed.\".equals(e.getMessage())) {\n    Throwable cause = e.getCause();   // the real diagnosis: parse error, RM unreachable, tokens...\n    log.error(\"distcp submission failed\", cause);\n  }\n}","preventionTips":["Smoke-test MR submission on both clusters before starting fedbalance.","Validate -bandwidth/-map values against distcp's accepted ranges in the job template.","Ensure delegation tokens for both nameservices exist in router-based federation."],"tags":["fedbalance","federation-balance","distcp","job-submission","yarn"],"backgroundTag":"job-submission-failed","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}