{"record":{"id":"654c2a5cda8ee706","repo":"apache/hadoop","slug":"could-not-contact-rm-after-milliseconds-654c2a","errorCode":null,"errorMessage":"Could not contact RM after {} milliseconds.","messagePattern":"Could not contact RM after (.+?) milliseconds\\.","errorType":"exception","errorClass":"RMContainerAllocationException","httpStatus":null,"severity":"critical","filePath":"hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java","lineNumber":829,"sourceCode":"      return null;\n    } catch (InvalidLabelResourceRequestException e) {\n      // If Invalid label exception is received means the requested label doesnt\n      // have access so killing job in this case.\n      String diagMsg = \"Requested node-label-expression is invalid: \"\n          + StringUtils.stringifyException(e);\n      LOG.info(diagMsg);\n      JobId jobId = this.getJob().getID();\n      eventHandler.handle(new JobDiagnosticsUpdateEvent(jobId, diagMsg));\n      eventHandler.handle(new JobEvent(jobId, JobEventType.JOB_KILL));\n      throw e;\n    } catch (Exception e) {\n      // This can happen when the connection to the RM has gone down. Keep\n      // re-trying until the retryInterval has expired.\n      if (System.currentTimeMillis() - retrystartTime >= retryInterval) {\n        LOG.error(\"Could not contact RM after \" + retryInterval + \" milliseconds.\");\n        eventHandler.handle(new JobEvent(this.getJob().getID(),\n                                         JobEventType.JOB_AM_REBOOT));\n        throw new RMContainerAllocationException(\"Could not contact RM after \" +\n                                retryInterval + \" milliseconds.\");\n      }\n      // Throw this up to the caller, which may decide to ignore it and\n      // continue to attempt to contact the RM.\n      throw e;\n    }\n    Resource newHeadRoom = getAvailableResources();\n    List<Container> newContainers = response.getAllocatedContainers();\n    // Setting NMTokens\n    if (response.getNMTokens() != null) {\n      for (NMToken nmToken : response.getNMTokens()) {\n        NMTokenCache.setNMToken(nmToken.getNodeId().toString(),\n            nmToken.getToken());\n      }\n    }\n\n    // Setting AMRMToken\n    if (response.getAMRMToken() != null) {","sourceCodeStart":811,"sourceCodeEnd":847,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java#L811-L847","documentation":"RMContainerAllocator's heartbeat catches a generic Exception from makeRemoteRequest and, once the elapsed time since the last successful allocate exceeds retryInterval (yarn.app.mapreduce.am.scheduler.retry.interval-ms, default 360000 ms), stops retrying: it logs this error, sends JOB_AM_REBOOT, and throws RMContainerAllocationException (without the underlying cause attached).","triggerScenarios":"Sustained RM RPC failures — RM down, network partition, scheduler RPC overloaded — persisting past the retry window; intermittent exceptions within the window are rethrown and retried instead.","commonSituations":"RM outage longer than the configured interval; wrong or stale RM address after failover in a non-HA client config; firewall drops between the AM's node and the RM; retryInterval shortened below real failover times.","solutions":["Restore/reach the RM and confirm the AM can resolve and dial yarn.resourcemanager.scheduler.address","Set up RM HA (yarn.resourcemanager.ha.enabled) so allocator retries switch to the standby","Increase yarn.app.mapreduce.am.scheduler.retry.interval-ms to cover expected failover windows","Resubmit the job after the AM reboots — it cannot resume without a live RM handshake"],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  runJob();\n} catch (RMContainerAllocationException e) {\n  if (e.getMessage().startsWith(\"Could not contact RM after\")) {\n    waitUntilRmReachable(); resubmit(jobConf);\n  } else { throw e; }\n}","preventionTips":["Configure RM HA and verify AMs receive both RM addresses","Size yarn.app.mapreduce.am.scheduler.retry.interval-ms above your real failover time","Check firewall/DNS between worker nodes and yarn.resourcemanager.scheduler.address before lowering the interval"],"tags":["mapreduce","yarn","resource-manager","connectivity","retry-timeout"],"backgroundTag":"resource-manager-unreachable","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}