{"record":{"id":"0878d1f3da566780","repo":"apache/hadoop","slug":"could-not-contact-rm-after-milliseconds","errorCode":null,"errorMessage":"Could not contact RM after {} milliseconds.","messagePattern":"Could not contact RM after (.+?) milliseconds\\.","errorType":"exception","errorClass":"YarnRuntimeException","httpStatus":null,"severity":"critical","filePath":"hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/local/LocalContainerAllocator.java","lineNumber":137,"sourceCode":"      // this application must clean itself up.\n      eventHandler.handle(new JobEvent(this.getJob().getID(),\n        JobEventType.JOB_AM_REBOOT));\n      throw new YarnRuntimeException(\n        \"Resource Manager doesn't recognize AttemptId: \"\n            + this.getContext().getApplicationID(), e);\n    } catch (ApplicationMasterNotRegisteredException e) {\n      LOG.info(\"ApplicationMaster is out of sync with ResourceManager,\"\n          + \" hence resync and send outstanding requests.\");\n      this.lastResponseID = 0;\n      register();\n    } catch (Exception e) {\n      // This can happen when the connection to the RM has gone down. Keep\n      // re-trying until the retryInterval has expired.\n      if (System.currentTimeMillis() - retrystartTime >= retryInterval) {\n        LOG.error(\"Could not contact RM after \" + retryInterval + \" milliseconds.\");\n        eventHandler.handle(new JobEvent(this.getJob().getID(),\n                                         JobEventType.INTERNAL_ERROR));\n        throw new YarnRuntimeException(\"Could not contact RM after \" +\n                                retryInterval + \" milliseconds.\");\n      }\n      // Throw this up to the caller, which may decide to ignore it and\n      // continue to attempt to contact the RM.\n      throw e;\n    }\n\n    if (allocateResponse != null) {\n      this.lastResponseID = allocateResponse.getResponseId();\n      Token token = allocateResponse.getAMRMToken();\n      if (token != null) {\n        updateAMRMToken(token);\n      }\n      Priority priorityFromResponse = Priority.newInstance(allocateResponse\n          .getApplicationPriority().getPriority());\n\n      // Update the job priority to Job directly.\n      getJob().setJobPriority(priorityFromResponse);","sourceCodeStart":119,"sourceCodeEnd":155,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/local/LocalContainerAllocator.java#L119-L155","documentation":"LocalContainerAllocator's allocate loop catches a generic Exception from the RM RPC and, once (now - retrystartTime) >= retryInterval (yarn.app.mapreduce.am.scheduler.retry.interval-ms, default 360000 ms), gives up: it logs this error, sends JobEventType.INTERNAL_ERROR, and throws YarnRuntimeException. Any earlier exceptions within the interval are rethrown to the caller for another retry round.","triggerScenarios":"RM unreachable for longer than the retry interval: RM process down, network partition between AM and RM, RM RPC throttling/queue full, repeated ConnectException/timeout on allocate.","commonSituations":"RM outage or failover storm outlasting the 6-minute default; misconfigured yarn.resourcemanager.scheduler.address (AM dialing wrong host); firewall/DNS issues in containerized or multi-NIC clusters; retryInterval lowered too aggressively.","solutions":["Check RM health and the AM-to-RM address (yarn.resourcemanager.scheduler.address) first — the AM log lists every underlying exception during the window","Enable/configure RM HA so the AM fails over instead of retrying a dead RM","Raise yarn.app.mapreduce.am.scheduler.retry.interval-ms if transient RM maintenance windows exceed the default","The job is failed via INTERNAL_ERROR — plan client-side resubmission once the RM is back"],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  runJob();\n} catch (YarnRuntimeException e) {\n  if (e.getMessage().startsWith(\"Could not contact RM after\")) {\n    waitUntilRmReachable(); // circuit-break, then resubmit\n    resubmit(jobConf);\n  } else { throw e; }\n}","preventionTips":["Deploy RM HA so allocator retries fail over instead of exhausting the interval","Keep yarn.app.mapreduce.am.scheduler.retry.interval-ms >= expected RM failover/maintenance windows","Alert on RM RPC error rates from AM logs to catch partitions before the interval expires"],"tags":["mapreduce","yarn","resource-manager","connectivity","retry-timeout"],"backgroundTag":"resource-manager-unreachable","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}