{"record":{"id":"b662592c8b6dbd5c","repo":"apache/hadoop","slug":"resource-manager-doesn-t-recognize-attemptid-b66259","errorCode":null,"errorMessage":"Resource Manager doesn't recognize AttemptId: {}","messagePattern":"Resource Manager doesn't recognize AttemptId: (.+?)","errorType":"exception","errorClass":"RMContainerAllocationException","httpStatus":null,"severity":"critical","filePath":"hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java","lineNumber":801,"sourceCode":"\n    // will be null the first time\n    Resource headRoom = Resources.clone(getAvailableResources());\n    AllocateResponse response;\n    /*\n     * If contact with RM is lost, the AM will wait MR_AM_TO_RM_WAIT_INTERVAL_MS\n     * milliseconds before aborting. During this interval, AM will still try\n     * to contact the RM.\n     */\n    try {\n      response = makeRemoteRequest();\n      // Reset retry count if no exception occurred.\n      retrystartTime = System.currentTimeMillis();\n    } catch (ApplicationAttemptNotFoundException e ) {\n      // This can happen if the RM has been restarted. If it is in that state,\n      // this application must clean itself up.\n      eventHandler.handle(new JobEvent(this.getJob().getID(),\n        JobEventType.JOB_AM_REBOOT));\n      throw new RMContainerAllocationException(\n        \"Resource Manager doesn't recognize AttemptId: \"\n            + this.getContext().getApplicationAttemptId(), e);\n    } catch (ApplicationMasterNotRegisteredException e) {\n      LOG.info(\"ApplicationMaster is out of sync with ResourceManager,\"\n          + \" hence resync and send outstanding requests.\");\n      // RM may have restarted, re-register with RM.\n      lastResponseID = 0;\n      register();\n      addOutstandingRequestOnResync();\n      return null;\n    } catch (InvalidLabelResourceRequestException e) {\n      // If Invalid label exception is received means the requested label doesnt\n      // have access so killing job in this case.\n      String diagMsg = \"Requested node-label-expression is invalid: \"\n          + StringUtils.stringifyException(e);\n      LOG.info(diagMsg);\n      JobId jobId = this.getJob().getID();\n      eventHandler.handle(new JobDiagnosticsUpdateEvent(jobId, diagMsg));","sourceCodeStart":783,"sourceCodeEnd":819,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java#L783-L819","documentation":"RMContainerAllocator (the normal, container-requesting allocator) hits the same condition as the local one: scheduler.allocate throws ApplicationAttemptNotFoundException because the RM lost this attempt (restart without work-preserving recovery, or the attempt was evicted). The AM requests JOB_AM_REBOOT so the job tears itself down, then throws RMContainerAllocationException wrapping the RM's cause.","triggerScenarios":"RM restart/failover with recovery disabled or failed while the MR job is mid-flight; RM state store lost; attempt expired at the RM (e.g., after long GC pause or clock issues) while the AM keeps allocating.","commonSituations":"Production RM restarts without a working state store (ZK/HDFS levelDB missing); RM HA failover where recovery is not enabled; long-running jobs during cluster upgrades.","solutions":["Configure RM restart properly: yarn.resourcemanager.recovery.enabled=true plus a state store, and yarn.resourcemanager.work-preserving-recovery.enabled=true","Resubmit the job — JOB_AM_REBOOT means this attempt cannot continue against the fresh RM","Verify RM logs around the failure time for the attempt eviction reason (AM expire, state loss)"],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  job.waitForCompletion(false);\n} catch (RMContainerAllocationException e) {\n  if (e.getMessage().contains(\"doesn't recognize AttemptId\")) {\n    resubmit(jobConf); // RM forgot the attempt (restart without recovery)\n  } else { throw e; }\n}","preventionTips":["Enable yarn.resourcemanager.recovery.enabled with a state store and work-preserving recovery on production RMs","Make submission idempotent (fresh output dirs or overwrite policy) so AM reboot triggers a clean rerun","Correlate the AM reboot timestamp with RM restart logs to confirm root cause"],"tags":["mapreduce","yarn","resource-manager","rm-restart","am-reboot"],"backgroundTag":"application-attempt-not-found","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}