{"record":{"id":"ca3c4221e4ba0fef","repo":"apache/hadoop","slug":"failed-to-run-job-diagnostics","errorCode":null,"errorMessage":"Failed to run job : {diagnostics}","messagePattern":"Failed to run job : (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java","lineNumber":341,"sourceCode":"    addHistoryToken(ts);\n\n    ApplicationSubmissionContext appContext =\n      createApplicationSubmissionContext(conf, jobSubmitDir, ts);\n\n    // Submit to ResourceManager\n    try {\n      ApplicationId applicationId =\n          resMgrDelegate.submitApplication(appContext);\n\n      ApplicationReport appMaster = resMgrDelegate\n          .getApplicationReport(applicationId);\n      String diagnostics =\n          (appMaster == null ?\n              \"application report is null\" : appMaster.getDiagnostics());\n      if (appMaster == null\n          || appMaster.getYarnApplicationState() == YarnApplicationState.FAILED\n          || appMaster.getYarnApplicationState() == YarnApplicationState.KILLED) {\n        throw new IOException(\"Failed to run job : \" +\n            diagnostics);\n      }\n      return clientCache.getClient(jobId).getJobStatus(jobId);\n    } catch (YarnException e) {\n      throw new IOException(e);\n    }\n  }\n\n  private LocalResource createApplicationResource(FileContext fs, Path p,\n      LocalResourceType type) throws IOException {\n    return createApplicationResource(fs, p, null, type,\n        LocalResourceVisibility.APPLICATION, false);\n  }\n\n  private LocalResource createApplicationResource(FileContext fs, Path p,\n      String fileSymlink, LocalResourceType type, LocalResourceVisibility viz,\n      Boolean uploadToSharedCache) throws IOException {\n    LocalResource rsrc = recordFactory.newRecordInstance(LocalResource.class);","sourceCodeStart":323,"sourceCodeEnd":359,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java#L323-L359","documentation":"Thrown by YARNRunner.submitJob after the ResourceManager accepted the submission but the immediately fetched ApplicationReport is null or shows YarnApplicationState FAILED/KILLED. The IOException embeds the report's diagnostics so the submitter sees why the application died at RM level. This is a genuine job-launch failure reported synchronously instead of through the job monitor.","triggerScenarios":"Submitting a job whose application is instantly FAILED (invalid queue, queue ACL denial, malformed app context, AM resource beyond limits) or KILLED (yarn.resourcemanager maximum-application timeout, admin kill racing submission, AM container launch failure detected immediately).","commonSituations":"mapreduce.job.queuename points to a nonexistent queue; AM container size (mapreduce.map.memory.mb / am-resource settings) exceeds queue or cluster max; Submission racing an RM restart/failover so the report comes back FAILED; Missing/staging permission errors on the job submit directory surfacing as instant failure","solutions":["Read the embedded diagnostics in the exception message — it carries the RM's exact reason","Validate queue name and AM resource requests against YARN configuration (yarn.app.mapreduce.am.resource.mb, queue capacities) before submit","Verify the staging directory (yarn.app.mapreduce.cli-staging-dir / mapreduce.job.submitter) is writable on the default FS","If caused by RM failover, resubmit after the RM is active and stable"],"exampleFix":"// before\njob.submit(); // IOException: Failed to run job : application ... failed\n\n// after\njob.getConfiguration().set(\"mapreduce.job.queuename\", \"existingQueue\");\ntry { job.submit(); }\ncatch (IOException e) {\n  // e.getMessage() contains RM diagnostics - surface them to the operator\n  throw new RuntimeException(\"submit failed: \" + e.getMessage(), e);\n}","handlingStrategy":"try-catch","validationCode":"Configuration conf = new Configuration();\nString queue = conf.get(\"mapreduce.job.queuename\", \"default\");\nif (!knownQueues(yarnClient).contains(queue))\n  throw new IllegalArgumentException(\"queue does not exist: \" + queue);\nint amMb = conf.getInt(\"yarn.app.mapreduce.am.resource.mb\", 1536);\nif (amMb > queueMaxMb(yarnClient, queue))\n  throw new IllegalArgumentException(\"AM memory above queue max\");","typeGuard":null,"tryCatchPattern":"try {\n  job.submit();\n} catch (IOException e) {\n  if (e.getMessage().startsWith(\"Failed to run job\")) {\n    // message embeds RM diagnostics — parse/present them, decide retry vs abort\n    reportSubmitFailure(e.getMessage());\n  } else throw e;\n}","preventionTips":["Validate queue name, queue ACLs, and AM resource limits before submitting","Always surface the embedded diagnostics — they name the real RM-side cause","Retry submissions that fail across RM failover windows; abort cleanly on ACL/resource denials","Verify the staging dir exists and is writable by the submitting user"],"tags":["hadoop","mapreduce","yarn","job-submission","application-failed","diagnostics"],"backgroundTag":"job-submission-failed","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}