apache/hadoop · error · IOException

Failed to run job : {diagnostics}

Error message

Failed to run job : {diagnostics}

What it means

Thrown by YARNRunner.submitJob after the ResourceManager accepted the submission but the immediately fetched ApplicationReport is null or shows YarnApplicationState FAILED/KILLED. The IOException embeds the report's diagnostics so the submitter sees why the application died at RM level. This is a genuine job-launch failure reported synchronously instead of through the job monitor.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java:341

    addHistoryToken(ts);

    ApplicationSubmissionContext appContext =
      createApplicationSubmissionContext(conf, jobSubmitDir, ts);

    // Submit to ResourceManager
    try {
      ApplicationId applicationId =
          resMgrDelegate.submitApplication(appContext);

      ApplicationReport appMaster = resMgrDelegate
          .getApplicationReport(applicationId);
      String diagnostics =
          (appMaster == null ?
              "application report is null" : appMaster.getDiagnostics());
      if (appMaster == null
          || appMaster.getYarnApplicationState() == YarnApplicationState.FAILED
          || appMaster.getYarnApplicationState() == YarnApplicationState.KILLED) {
        throw new IOException("Failed to run job : " +
            diagnostics);
      }
      return clientCache.getClient(jobId).getJobStatus(jobId);
    } catch (YarnException e) {
      throw new IOException(e);
    }
  }

  private LocalResource createApplicationResource(FileContext fs, Path p,
      LocalResourceType type) throws IOException {
    return createApplicationResource(fs, p, null, type,
        LocalResourceVisibility.APPLICATION, false);
  }

  private LocalResource createApplicationResource(FileContext fs, Path p,
      String fileSymlink, LocalResourceType type, LocalResourceVisibility viz,
      Boolean uploadToSharedCache) throws IOException {
    LocalResource rsrc = recordFactory.newRecordInstance(LocalResource.class);

View on GitHub (pinned to 2add963021)

Solutions

1. Read the embedded diagnostics in the exception message — it carries the RM's exact reason
2. Validate queue name and AM resource requests against YARN configuration (yarn.app.mapreduce.am.resource.mb, queue capacities) before submit
3. Verify the staging directory (yarn.app.mapreduce.cli-staging-dir / mapreduce.job.submitter) is writable on the default FS
4. If caused by RM failover, resubmit after the RM is active and stable

Example fix

// before
job.submit(); // IOException: Failed to run job : application ... failed

// after
job.getConfiguration().set("mapreduce.job.queuename", "existingQueue");
try { job.submit(); }
catch (IOException e) {
  // e.getMessage() contains RM diagnostics - surface them to the operator
  throw new RuntimeException("submit failed: " + e.getMessage(), e);
}

Configuration conf = new Configuration();
String queue = conf.get("mapreduce.job.queuename", "default");
if (!knownQueues(yarnClient).contains(queue))
  throw new IllegalArgumentException("queue does not exist: " + queue);
int amMb = conf.getInt("yarn.app.mapreduce.am.resource.mb", 1536);
if (amMb > queueMaxMb(yarnClient, queue))
  throw new IllegalArgumentException("AM memory above queue max");

try {
  job.submit();
} catch (IOException e) {
  if (e.getMessage().startsWith("Failed to run job")) {
    // message embeds RM diagnostics — parse/present them, decide retry vs abort
    reportSubmitFailure(e.getMessage());
  } else throw e;
}

When it happens

Trigger: Submitting a job whose application is instantly FAILED (invalid queue, queue ACL denial, malformed app context, AM resource beyond limits) or KILLED (yarn.resourcemanager maximum-application timeout, admin kill racing submission, AM container launch failure detected immediately).

Common situations: mapreduce.job.queuename points to a nonexistent queue; AM container size (mapreduce.map.memory.mb / am-resource settings) exceeds queue or cluster max; Submission racing an RM restart/failover so the report comes back FAILED; Missing/staging permission errors on the job submit directory surfacing as instant failure

Related errors

• Unrecognized task type: {}
• Unrecognized State: {}
• Unrecognized Phase: {}
• Unrecognized status: {}
• Unrecognized job state: {}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/ca3c4221e4ba0fef. Report an issue: GitHub.