apache/hadoop · error · IOException

Unexpected ZooKeeper issue fetching active node info

Error message

Unexpected ZooKeeper issue fetching active node info

What it means

ZKFailoverController.getCurrentActive() reads the activeBreadCrumb znode via elector.getActiveData() while servicing a graceful failover. ActiveNotFoundException is treated as 'no active' and returns null, but any other org.apache.zookeeper.KeeperException is wrapped in an IOException with this message. It signals a ZooKeeper-level problem (connectivity, session, or ACL) rather than a failover-logic problem.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/ZKFailoverController.java:797

  }

  /**
   * @return an {@link HAServiceTarget} for the current active node
   * in the cluster, or null if no node is active.
   * @throws IOException if a ZK-related issue occurs
   * @throws InterruptedException if thread is interrupted 
   */
  private HAServiceTarget getCurrentActive()
      throws IOException, InterruptedException {
    synchronized (elector) {
      synchronized (this) {
        byte[] activeData;
        try {
          activeData = elector.getActiveData();
        } catch (ActiveNotFoundException e) {
          return null;
        } catch (KeeperException ke) {
          throw new IOException(
              "Unexpected ZooKeeper issue fetching active node info", ke);
        }
        
        HAServiceTarget oldActive = dataToTarget(activeData);
        return oldActive;
      }
    }
  }

  /**
   * Check the current state of the service, and join the election
   * if it should be in the election.
   */
  private void recheckElectability() {
    // Maintain lock ordering of elector -> ZKFC
    synchronized (elector) {
      synchronized (this) {
        boolean healthy = lastHealthState == State.SERVICE_HEALTHY;

View on GitHub (pinned to 2add963021)

Solutions

1. Check ZooKeeper health first: 'echo ruok | nc <zk-host> 2181', quorum status via 'zkServer.sh status', and ZK server logs
2. Verify every ZKFC uses identical ha.zookeeper.quorum and zookeeper.auth settings in hdfs-site.xml/core-site.xml
3. After ZK recovers, ZKFCs rejoin the election automatically; retry 'hdfs haadmin -failover'
4. If sessions expire repeatedly, raise ha.zookeeper.session-timeout-ms (and matching zookeeper.session.timeout)

// before failover, verify the ZKFC's elector is connected
echo stat | nc <zk-host> 2181   # ensemble reachable
# and check the local ZKFC is running / its ZK session:
jps | grep DFSZKFailoverController

try {
  HAServiceTarget t = getCurrentActiveEquivalent(); // gracefulFailover path
} catch (IOException e) {
  if (e.getCause() instanceof KeeperException) {
    // transient ZK condition: back off and re-check connectivity, then retry
  }
}

When it happens

Trigger: Graceful failover while the ZKFC's ZooKeeper connection is broken (ConnectionLoss/SessionExpired), ACL mismatches on /hadoop-ha/<nameservice> when zookeeper.auth digest settings differ between ZKFCs, or the ZK quorum itself being down or split.

Common situations: ZooKeeper ensemble outage or rolling restart during failover drills; misconfigured ha.zookeeper.{quorum,auth,session-timeout-ms}; ZK znode limits or disk full on ZooKeeper servers causing errors.

Related errors

• {} is in observer state. Cannot be failover target
• Got an IO exception
• ZK Failover Controller failed: {}
• Serverside implements {}. The following requested protocol i
• Manual HA control for this NameNode is disallowed, because a

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/f826d0a3023ea3c7. Report an issue: GitHub.