{"record":{"id":"3223cd6e77fb8913","repo":"apache/hadoop","slug":"unable-to-become-active-local-node-did-not-get-an","errorCode":null,"errorMessage":"Unable to become active. Local node did not get an opportunity to do so from ZooKeeper, or the local node took too long to transition to active.","messagePattern":"Unable to become active\\. Local node did not get an opportunity to do so from ZooKeeper, or the local node took too long to transition to active\\.","errorType":"exception","errorClass":"ServiceFailedException","httpStatus":null,"severity":"error","filePath":"hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/ZKFailoverController.java","lineNumber":723,"sourceCode":"      activeNode != null : \"Active node does not match any known remote node\";\n\n    // Phase 3b: ask the old active to yield\n    otherZkfcs.add(cedeRemoteActive(activeNode, timeout));\n\n    // Phase 4: wait for the normal election to make the local node\n    // active.\n    ActiveAttemptRecord attempt = waitForActiveAttempt(timeout + 60000, st);\n    \n    if (attempt == null) {\n      // We didn't even make an attempt to become active.\n      synchronized(this) {\n        if (lastHealthState != State.SERVICE_HEALTHY) {\n          throw new ServiceFailedException(\"Unable to become active. \" +\n            \"Service became unhealthy while trying to failover.\");          \n        }\n      }\n      \n      throw new ServiceFailedException(\"Unable to become active. \" +\n          \"Local node did not get an opportunity to do so from ZooKeeper, \" +\n          \"or the local node took too long to transition to active.\");\n    }\n\n    // Phase 5. At this point, we made some attempt to become active. So we\n    // can tell the old active to rejoin if it wants. This allows a quick\n    // fail-back if we immediately crash.\n    for (ZKFCProtocol zkfc : otherZkfcs) {\n      zkfc.cedeActive(-1);\n    }\n\n    if (attempt.succeeded) {\n      LOG.info(\"Successfully became active. \" + attempt.status);\n    } else {\n      // Propagate failure\n      String msg = \"Failed to become active. \" + attempt.status;\n      throw new ServiceFailedException(msg);\n    }","sourceCodeStart":705,"sourceCodeEnd":741,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/ZKFailoverController.java#L705-L741","documentation":"gracefulFailover() Phase 4: waitForActiveAttempt(timeout+60000) returned null while health stayed SERVICE_HEALTHY — the local ZKFC never got the opportunity to become active from ZooKeeper, or the transition took longer than the window; thrown as ServiceFailedException('Unable to become active. Local node did not get an opportunity to do so from ZooKeeper, or the local node took too long to transition to active.').","triggerScenarios":"The old active was asked to cede, but the local ZKFC never entered/won the election and never called becomeActive within timeout+60s: ZooKeeper latency or session issues, old active's ZKFC not actually ceding, or the local NameNode's transitionToActive exceeding the window (e.g. long checkpoint on transition).","commonSituations":"Overloaded/high-latency ZooKeeper ensemble; ha.zookeeper.session-timeout-ms too large so re-election is slow; old active ZKFC stuck; local NN slow to become active because it must checkpoint a large edit tail; timeout passed to gracefulFailover too small.","solutions":["Check ZooKeeper health/latency and ZKFC logs on both nodes — did the local ZKFC rejoin the election after the cede?","Confirm the old active actually transitioned to standby (its logs show cedeActive handling); if it never ceded, fix its ZKFC.","Address local transition slowness: keep the standby's checkpoint state current, check JournalNode performance, look for a long tail of edits to replay.","Retry the failover with a larger timeout."],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// Before failover: verify ZK latency is sane and old active ZKFC is responsive\n// zkCli.sh -server zk:2181 create /probe x && delete /probe  (time it)\n// hdfs haadmin -getServiceState on the old active must answer quickly","typeGuard":null,"tryCatchPattern":"try {\n  zkfc.gracefulFailover();\n} catch (ServiceFailedException e) {\n  if (e.getMessage().contains(\"did not get an opportunity\")) {\n    // election never reached local node: check ZK latency + old ZKFC cede,\n    // then retry with a larger timeout after fixing the slow component\n  }\n}","preventionTips":["Monitor ZooKeeper latency; slow ensembles stretch every graceful failover phase.","Tune ha.zookeeper.session-timeout-ms vs expected ZK recovery time on your ensemble.","Keep standby checkpoint currency high so transitionToActive is quick; slow transitions are the classic cause of this timeout."],"tags":["hadoop","high-availability","graceful-failover","zookeeper","timeout","election"],"backgroundTag":"election-timeout","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}