{"record":{"id":"f826d0a3023ea3c7","repo":"apache/hadoop","slug":"unexpected-zookeeper-issue-fetching-active-node-in","errorCode":null,"errorMessage":"Unexpected ZooKeeper issue fetching active node info","messagePattern":"Unexpected ZooKeeper issue fetching active node info","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/ZKFailoverController.java","lineNumber":797,"sourceCode":"  }\n\n  /**\n   * @return an {@link HAServiceTarget} for the current active node\n   * in the cluster, or null if no node is active.\n   * @throws IOException if a ZK-related issue occurs\n   * @throws InterruptedException if thread is interrupted \n   */\n  private HAServiceTarget getCurrentActive()\n      throws IOException, InterruptedException {\n    synchronized (elector) {\n      synchronized (this) {\n        byte[] activeData;\n        try {\n          activeData = elector.getActiveData();\n        } catch (ActiveNotFoundException e) {\n          return null;\n        } catch (KeeperException ke) {\n          throw new IOException(\n              \"Unexpected ZooKeeper issue fetching active node info\", ke);\n        }\n        \n        HAServiceTarget oldActive = dataToTarget(activeData);\n        return oldActive;\n      }\n    }\n  }\n\n  /**\n   * Check the current state of the service, and join the election\n   * if it should be in the election.\n   */\n  private void recheckElectability() {\n    // Maintain lock ordering of elector -> ZKFC\n    synchronized (elector) {\n      synchronized (this) {\n        boolean healthy = lastHealthState == State.SERVICE_HEALTHY;","sourceCodeStart":779,"sourceCodeEnd":815,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/ZKFailoverController.java#L779-L815","documentation":"ZKFailoverController.getCurrentActive() reads the activeBreadCrumb znode via elector.getActiveData() while servicing a graceful failover. ActiveNotFoundException is treated as 'no active' and returns null, but any other org.apache.zookeeper.KeeperException is wrapped in an IOException with this message. It signals a ZooKeeper-level problem (connectivity, session, or ACL) rather than a failover-logic problem.","triggerScenarios":"Graceful failover while the ZKFC's ZooKeeper connection is broken (ConnectionLoss/SessionExpired), ACL mismatches on /hadoop-ha/<nameservice> when zookeeper.auth digest settings differ between ZKFCs, or the ZK quorum itself being down or split.","commonSituations":"ZooKeeper ensemble outage or rolling restart during failover drills; misconfigured ha.zookeeper.{quorum,auth,session-timeout-ms}; ZK znode limits or disk full on ZooKeeper servers causing errors.","solutions":["Check ZooKeeper health first: 'echo ruok | nc <zk-host> 2181', quorum status via 'zkServer.sh status', and ZK server logs","Verify every ZKFC uses identical ha.zookeeper.quorum and zookeeper.auth settings in hdfs-site.xml/core-site.xml","After ZK recovers, ZKFCs rejoin the election automatically; retry 'hdfs haadmin -failover'","If sessions expire repeatedly, raise ha.zookeeper.session-timeout-ms (and matching zookeeper.session.timeout)"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// before failover, verify the ZKFC's elector is connected\necho stat | nc <zk-host> 2181   # ensemble reachable\n# and check the local ZKFC is running / its ZK session:\njps | grep DFSZKFailoverController","typeGuard":null,"tryCatchPattern":"try {\n  HAServiceTarget t = getCurrentActiveEquivalent(); // gracefulFailover path\n} catch (IOException e) {\n  if (e.getCause() instanceof KeeperException) {\n    // transient ZK condition: back off and re-check connectivity, then retry\n  }\n}","preventionTips":["Monitor ZooKeeper connection/session metrics exposed by the ZKFC JMX","Keep ha.zookeeper.quorum and zookeeper.auth identical across all ZKFCs","Size ha.zookeeper.session-timeout-ms to survive short ZK/GC pauses"],"tags":["hadoop","ha","zookeeper","zkfc","failover","network"],"backgroundTag":"zookeeper-connection-loss","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}