What causes "Could not contact RM after {} milliseconds." in apache/hadoop?

Check RM health and the AM-to-RM address (yarn.resourcemanager.scheduler.address) first — the AM log lists every underlying exception during the window

apache/hadoop · critical · YarnRuntimeException

Could not contact RM after {} milliseconds.

Error message

Could not contact RM after {} milliseconds.

What it means

LocalContainerAllocator's allocate loop catches a generic Exception from the RM RPC and, once (now - retrystartTime) >= retryInterval (yarn.app.mapreduce.am.scheduler.retry.interval-ms, default 360000 ms), gives up: it logs this error, sends JobEventType.INTERNAL_ERROR, and throws YarnRuntimeException. Any earlier exceptions within the interval are rethrown to the caller for another retry round.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/local/LocalContainerAllocator.java:137

      // this application must clean itself up.
      eventHandler.handle(new JobEvent(this.getJob().getID(),
        JobEventType.JOB_AM_REBOOT));
      throw new YarnRuntimeException(
        "Resource Manager doesn't recognize AttemptId: "
            + this.getContext().getApplicationID(), e);
    } catch (ApplicationMasterNotRegisteredException e) {
      LOG.info("ApplicationMaster is out of sync with ResourceManager,"
          + " hence resync and send outstanding requests.");
      this.lastResponseID = 0;
      register();
    } catch (Exception e) {
      // This can happen when the connection to the RM has gone down. Keep
      // re-trying until the retryInterval has expired.
      if (System.currentTimeMillis() - retrystartTime >= retryInterval) {
        LOG.error("Could not contact RM after " + retryInterval + " milliseconds.");
        eventHandler.handle(new JobEvent(this.getJob().getID(),
                                         JobEventType.INTERNAL_ERROR));
        throw new YarnRuntimeException("Could not contact RM after " +
                                retryInterval + " milliseconds.");
      }
      // Throw this up to the caller, which may decide to ignore it and
      // continue to attempt to contact the RM.
      throw e;
    }

    if (allocateResponse != null) {
      this.lastResponseID = allocateResponse.getResponseId();
      Token token = allocateResponse.getAMRMToken();
      if (token != null) {
        updateAMRMToken(token);
      }
      Priority priorityFromResponse = Priority.newInstance(allocateResponse
          .getApplicationPriority().getPriority());

      // Update the job priority to Job directly.
      getJob().setJobPriority(priorityFromResponse);

View on GitHub (pinned to 2add963021)

Solutions

Check RM health and the AM-to-RM address (yarn.resourcemanager.scheduler.address) first — the AM log lists every underlying exception during the window
Enable/configure RM HA so the AM fails over instead of retrying a dead RM
Raise yarn.app.mapreduce.am.scheduler.retry.interval-ms if transient RM maintenance windows exceed the default
The job is failed via INTERNAL_ERROR — plan client-side resubmission once the RM is back

Defensive patterns

Strategy: retry

Try / catch

try {
  runJob();
} catch (YarnRuntimeException e) {
  if (e.getMessage().startsWith("Could not contact RM after")) {
    waitUntilRmReachable(); // circuit-break, then resubmit
    resubmit(jobConf);
  } else { throw e; }
}

Prevention

Deploy RM HA so allocator retries fail over instead of exhausting the interval
Keep yarn.app.mapreduce.am.scheduler.retry.interval-ms >= expected RM failover/maintenance windows
Alert on RM RPC error rates from AM logs to catch partitions before the interval expires

When it happens

Trigger: RM unreachable for longer than the retry interval: RM process down, network partition between AM and RM, RM RPC throttling/queue full, repeated ConnectException/timeout on allocate.

Common situations: RM outage or failover storm outlasting the 6-minute default; misconfigured yarn.resourcemanager.scheduler.address (AM dialing wrong host); firewall/DNS issues in containerized or multi-NIC clusters; retryInterval lowered too aggressively.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/0878d1f3da566780. Report an issue: GitHub.