apache/hadoop · critical · RMContainerAllocationException

Could not contact RM after {} milliseconds.

Error message

Could not contact RM after {} milliseconds.

What it means

RMContainerAllocator's heartbeat catches a generic Exception from makeRemoteRequest and, once the elapsed time since the last successful allocate exceeds retryInterval (yarn.app.mapreduce.am.scheduler.retry.interval-ms, default 360000 ms), stops retrying: it logs this error, sends JOB_AM_REBOOT, and throws RMContainerAllocationException (without the underlying cause attached).

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java:829

      return null;
    } catch (InvalidLabelResourceRequestException e) {
      // If Invalid label exception is received means the requested label doesnt
      // have access so killing job in this case.
      String diagMsg = "Requested node-label-expression is invalid: "
          + StringUtils.stringifyException(e);
      LOG.info(diagMsg);
      JobId jobId = this.getJob().getID();
      eventHandler.handle(new JobDiagnosticsUpdateEvent(jobId, diagMsg));
      eventHandler.handle(new JobEvent(jobId, JobEventType.JOB_KILL));
      throw e;
    } catch (Exception e) {
      // This can happen when the connection to the RM has gone down. Keep
      // re-trying until the retryInterval has expired.
      if (System.currentTimeMillis() - retrystartTime >= retryInterval) {
        LOG.error("Could not contact RM after " + retryInterval + " milliseconds.");
        eventHandler.handle(new JobEvent(this.getJob().getID(),
                                         JobEventType.JOB_AM_REBOOT));
        throw new RMContainerAllocationException("Could not contact RM after " +
                                retryInterval + " milliseconds.");
      }
      // Throw this up to the caller, which may decide to ignore it and
      // continue to attempt to contact the RM.
      throw e;
    }
    Resource newHeadRoom = getAvailableResources();
    List<Container> newContainers = response.getAllocatedContainers();
    // Setting NMTokens
    if (response.getNMTokens() != null) {
      for (NMToken nmToken : response.getNMTokens()) {
        NMTokenCache.setNMToken(nmToken.getNodeId().toString(),
            nmToken.getToken());
      }
    }

    // Setting AMRMToken
    if (response.getAMRMToken() != null) {

View on GitHub (pinned to 2add963021)

Solutions

1. Restore/reach the RM and confirm the AM can resolve and dial yarn.resourcemanager.scheduler.address
2. Set up RM HA (yarn.resourcemanager.ha.enabled) so allocator retries switch to the standby
3. Increase yarn.app.mapreduce.am.scheduler.retry.interval-ms to cover expected failover windows
4. Resubmit the job after the AM reboots — it cannot resume without a live RM handshake

try {
  runJob();
} catch (RMContainerAllocationException e) {
  if (e.getMessage().startsWith("Could not contact RM after")) {
    waitUntilRmReachable(); resubmit(jobConf);
  } else { throw e; }
}

When it happens

Trigger: Sustained RM RPC failures — RM down, network partition, scheduler RPC overloaded — persisting past the retry window; intermittent exceptions within the window are rethrown and retried instead.

Common situations: RM outage longer than the configured interval; wrong or stale RM address after failover in a non-HA client config; firewall drops between the AM's node and the RM; retryInterval shortened below real failover times.

Related errors

• Could not contact RM after {} milliseconds.
• Resource Manager doesn't recognize AttemptId: {}
• Resource Manager doesn't recognize AttemptId: {}
• "Mkdirs failed to create " + reduceIn.getParent().toString()
• "Couldn't rename " + mapOut

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/654c2a5cda8ee706. Report an issue: GitHub.