apache/hadoop · critical · RMContainerAllocationException

Resource Manager doesn't recognize AttemptId: {}

Error message

Resource Manager doesn't recognize AttemptId: {}

What it means

RMContainerAllocator (the normal, container-requesting allocator) hits the same condition as the local one: scheduler.allocate throws ApplicationAttemptNotFoundException because the RM lost this attempt (restart without work-preserving recovery, or the attempt was evicted). The AM requests JOB_AM_REBOOT so the job tears itself down, then throws RMContainerAllocationException wrapping the RM's cause.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java:801

    // will be null the first time
    Resource headRoom = Resources.clone(getAvailableResources());
    AllocateResponse response;
    /*
     * If contact with RM is lost, the AM will wait MR_AM_TO_RM_WAIT_INTERVAL_MS
     * milliseconds before aborting. During this interval, AM will still try
     * to contact the RM.
     */
    try {
      response = makeRemoteRequest();
      // Reset retry count if no exception occurred.
      retrystartTime = System.currentTimeMillis();
    } catch (ApplicationAttemptNotFoundException e ) {
      // This can happen if the RM has been restarted. If it is in that state,
      // this application must clean itself up.
      eventHandler.handle(new JobEvent(this.getJob().getID(),
        JobEventType.JOB_AM_REBOOT));
      throw new RMContainerAllocationException(
        "Resource Manager doesn't recognize AttemptId: "
            + this.getContext().getApplicationAttemptId(), e);
    } catch (ApplicationMasterNotRegisteredException e) {
      LOG.info("ApplicationMaster is out of sync with ResourceManager,"
          + " hence resync and send outstanding requests.");
      // RM may have restarted, re-register with RM.
      lastResponseID = 0;
      register();
      addOutstandingRequestOnResync();
      return null;
    } catch (InvalidLabelResourceRequestException e) {
      // If Invalid label exception is received means the requested label doesnt
      // have access so killing job in this case.
      String diagMsg = "Requested node-label-expression is invalid: "
          + StringUtils.stringifyException(e);
      LOG.info(diagMsg);
      JobId jobId = this.getJob().getID();
      eventHandler.handle(new JobDiagnosticsUpdateEvent(jobId, diagMsg));

View on GitHub (pinned to 2add963021)

Solutions

1. Configure RM restart properly: yarn.resourcemanager.recovery.enabled=true plus a state store, and yarn.resourcemanager.work-preserving-recovery.enabled=true
2. Resubmit the job — JOB_AM_REBOOT means this attempt cannot continue against the fresh RM
3. Verify RM logs around the failure time for the attempt eviction reason (AM expire, state loss)

try {
  job.waitForCompletion(false);
} catch (RMContainerAllocationException e) {
  if (e.getMessage().contains("doesn't recognize AttemptId")) {
    resubmit(jobConf); // RM forgot the attempt (restart without recovery)
  } else { throw e; }
}

When it happens

Trigger: RM restart/failover with recovery disabled or failed while the MR job is mid-flight; RM state store lost; attempt expired at the RM (e.g., after long GC pause or clock issues) while the AM keeps allocating.

Common situations: Production RM restarts without a working state store (ZK/HDFS levelDB missing); RM HA failover where recovery is not enabled; long-running jobs during cluster upgrades.

Related errors

• Resource Manager doesn't recognize AttemptId: {}
• Could not contact RM after {} milliseconds.
• Could not contact RM after {} milliseconds.
• "Mkdirs failed to create " + reduceIn.getParent().toString()
• "Couldn't rename " + mapOut

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/b662592c8b6dbd5c. Report an issue: GitHub.