apache/hadoop · error · IOException

Submit job failed.

Error message

Submit job failed.

What it means

submitDistCpJob() wraps the entire distcp submission - OptionsParser.parse() of the assembled command, constructing the DistCp object, and createAndSubmitJob() - in a single IOException('Submit job failed.', e). The meaningful diagnostic is always the chained cause; the wrapper message alone carries no information. It fires before any mapper runs, so no data has been copied when you see it.

Source

Thrown at hadoop-tools/hadoop-federation-balance/src/main/java/org/apache/hadoop/tools/fedbalance/DistCpProcedure.java:504

    command.add(mapNum + "");
    command.add("-bandwidth");
    command.add(bandWidth + "");
    command.add(srcParam);
    command.add(dstParam);

    Configuration config = new Configuration(conf);
    DistCp distCp;
    try {
      distCp = new DistCp(config,
          OptionsParser.parse(command.toArray(new String[]{})));
      Job job = distCp.createAndSubmitJob();
      LOG.info("Submit distcp job={}", job);
      if (enabledForTest) {
        localJob = job;
      }
      return job.getJobID().toString();
    } catch (Exception e) {
      throw new IOException("Submit job failed.", e);
    }
  }

  @Override
  public void write(DataOutput out) throws IOException {
    super.write(out);
    context.write(out);
    if (jobId == null) {
      out.writeBoolean(false);
    } else {
      out.writeBoolean(true);
      Text.writeString(out, jobId);
    }
    out.writeInt(stage.ordinal());
    if (fPerm == null) {
      out.writeBoolean(false);
    } else {
      out.writeBoolean(true);

View on GitHub (pinned to 2add963021)

Solutions

1. Read the cause chain first: find 'Caused by:' under this exception in the log and fix that specific error.
2. Verify MR submission works at all: run a trivial 'hadoop distcp' or 'hadoop jar hadoop-mapreduce-examples' job on the same client.
3. Check the fedbalance parameters (map count, bandwidth) are within distcp's accepted ranges.
4. For cross-cluster runs, confirm delegation tokens / keytab for both nameservices are available to the client.

try {
  FedBalance.main(args);
} catch (IOException e) {
  if ("Submit job failed.".equals(e.getMessage())) {
    Throwable cause = e.getCause();   // the real diagnosis: parse error, RM unreachable, tokens...
    log.error("distcp submission failed", cause);
  }
}

When it happens

Trigger: Any exception from: (a) invalid distcp options assembled from fedbalance parameters (bad bandwidth or map count), (b) DistCp input validation rejecting src/dst options, or (c) MapReduce submission failing - ResourceManager unreachable, staging directory issues, missing delegation tokens in cross-cluster federation setups.

Common situations: Mistyped -bandwidth/-map values passed through FedBalanceOptions; YARN not running or rm address misconfigured; fs.defaultFS or mapreduce.jobtracker.staging.dir problems; router-based federation without tokens for the remote nameservice.

Related errors

• {src} should be a directory.
• {dst} already exists.
• {src} shouldn't enable snapshot.
• DistCp failed. jobId={jobId} failure={failureInfo}
• Final DistCp failed. Failure: {failureInfo}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/a4b44f458ced8dde. Report an issue: GitHub.