{"record":{"id":"c17023b529345e80","repo":"apache/hadoop","slug":"unable-to-become-active-service-became-unhealthy","errorCode":null,"errorMessage":"Unable to become active. Service became unhealthy while trying to failover.","messagePattern":"Unable to become active\\. Service became unhealthy while trying to failover\\.","errorType":"exception","errorClass":"ServiceFailedException","httpStatus":null,"severity":"error","filePath":"hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/ZKFailoverController.java","lineNumber":718,"sourceCode":"      }\n      otherZkfcs.add(cedeRemoteActive(remote, timeout));\n    }\n\n    assert\n      activeNode != null : \"Active node does not match any known remote node\";\n\n    // Phase 3b: ask the old active to yield\n    otherZkfcs.add(cedeRemoteActive(activeNode, timeout));\n\n    // Phase 4: wait for the normal election to make the local node\n    // active.\n    ActiveAttemptRecord attempt = waitForActiveAttempt(timeout + 60000, st);\n    \n    if (attempt == null) {\n      // We didn't even make an attempt to become active.\n      synchronized(this) {\n        if (lastHealthState != State.SERVICE_HEALTHY) {\n          throw new ServiceFailedException(\"Unable to become active. \" +\n            \"Service became unhealthy while trying to failover.\");          \n        }\n      }\n      \n      throw new ServiceFailedException(\"Unable to become active. \" +\n          \"Local node did not get an opportunity to do so from ZooKeeper, \" +\n          \"or the local node took too long to transition to active.\");\n    }\n\n    // Phase 5. At this point, we made some attempt to become active. So we\n    // can tell the old active to rejoin if it wants. This allows a quick\n    // fail-back if we immediately crash.\n    for (ZKFCProtocol zkfc : otherZkfcs) {\n      zkfc.cedeActive(-1);\n    }\n\n    if (attempt.succeeded) {\n      LOG.info(\"Successfully became active. \" + attempt.status);","sourceCodeStart":700,"sourceCodeEnd":736,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/ZKFailoverController.java#L700-L736","documentation":"In gracefulFailover() Phase 4, if waitForActiveAttempt returns no ActiveAttemptRecord within timeout+60000 ms and lastHealthState != SERVICE_HEALTHY, the ZKFC throws ServiceFailedException('Unable to become active. Service became unhealthy while trying to failover.') — the local service's health degraded during the takeover window.","triggerScenarios":"The local node's health monitor reported a state other than SERVICE_HEALTHY (health check RPC failed, checkHealth threw, monitor shut down) between the cede request to the old active and the wait for the local transition attempt.","commonSituations":"NameNode crashed or OOM-killed mid-failover; transient health-check RPC failures to the local NN; disk full or edits-dir errors turning the NN unhealthy exactly during failover; NN restarted during the operation.","solutions":["Check the local NameNode's health and logs for whatever made the health monitor leave SERVICE_HEALTHY (crash, OOM, disk, checkpoint failure).","Run 'hdfs haadmin -checkHealth <nnId>' once it responds.","Fix the health issue, then retry the graceful failover.","If the old active already ceded, the cluster has no active — verify with getAllServiceState and force/allow election."],"exampleFix":null,"handlingStrategy":"validation","validationCode":"// Pre-flight the target the same way the ZKFC does\nzkfc.checkEligibleForFailover(); // throws ServiceFailedException if not healthy/observer\n// or externally: hdfs haadmin -checkHealth <nnId>","typeGuard":null,"tryCatchPattern":"try {\n  zkfc.gracefulFailover();\n} catch (ServiceFailedException e) {\n  if (e.getMessage().contains(\"Service became unhealthy\")) {\n    // inspect target NN health/logs; do not retry until health is green;\n    // also confirm whether the old active needs to rejoin (cedeActive(-1))\n  }\n}","preventionTips":["Run 'hdfs haadmin -checkHealth' on the failover target before every manual failover.","Alert on health-monitor state changes on all HA nodes so you never failover into an unhealthy node.","Keep NN memory/disks healthy; most mid-failover health losses are OOM or disk-full events."],"tags":["hadoop","high-availability","graceful-failover","health-check"],"backgroundTag":"service-health-check-failed","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}