What causes "ZK Failover Controller failed: {}" in apache/hadoop?

Read the ZKFC log line 'Fatal error occurred: ' immediately before the exception — it names the exact condition.

apache/hadoop · critical · RuntimeException

ZK Failover Controller failed: {}

Error message

ZK Failover Controller failed: {}

What it means

ZKFailoverController.mainLoop() blocks until fatalError is set by fatalError(String) — called on unrecoverable conditions such as fatal ZooKeeper events or parent-znode creation failure — then throws RuntimeException('ZK Failover Controller failed: <err>'). It means the ZKFC process hit a condition it cannot recover from and is exiting.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/ZKFailoverController.java:401

        zkTimeout, getParentZnode(), zkAcls, zkAuths,
        new ElectorCallbacks(), maxRetryNum, truststoreKeystore);
  }
  
  private String getParentZnode() {
    String znode = conf.get(ZK_PARENT_ZNODE_KEY,
        ZK_PARENT_ZNODE_DEFAULT);
    if (!znode.endsWith("/")) {
      znode += "/";
    }
    return znode + getScopeInsideParentNode();
  }

  private synchronized void mainLoop() throws InterruptedException {
    while (fatalError == null) {
      wait();
    }
    assert fatalError != null; // only get here on fatal
    throw new RuntimeException(
        "ZK Failover Controller failed: " + fatalError);
  }
  
  private synchronized void fatalError(String err) {
    LOG.error("Fatal error occurred:" + err);
    fatalError = err;
    notifyAll();
  }
  
  private synchronized void becomeActive() throws ServiceFailedException {
    LOG.info("Trying to make " + localTarget + " active...");
    try {
      HAServiceProtocolHelper.transitionToActive(localTarget.getProxy(
          conf, FailoverController.getRpcTimeoutToNewActive(conf)),
          createReqInfo());
      String msg = "Successfully transitioned " + localTarget +
          " to active state";
      LOG.info(msg);

View on GitHub (pinned to 2add963021)

Solutions

Read the ZKFC log line 'Fatal error occurred:<err>' immediately before the exception — it names the exact condition.
Verify ZooKeeper health: zkCli.sh (or 'echo ruok | nc zk 2181') against every host in ha.zookeeper.quorum and check client port reachability.
Fix znode permissions/ownership, or re-create the parent znode with 'hdfs zkfc -formatZK -force' (destroys HA state — use with caution) after fixing ACLs/auth.
Restart the ZKFC daemon once the underlying cause is fixed.

Defensive patterns

Strategy: try-catch

Validate before calling

// Before starting / during ops: verify the ZK quorum and parent znode
// from the ZKFC host:
//   zkCli.sh -server zk1:2181 ls /hadoop-ha
// Any ConnectTimeout/NoNode/ACL failure here predicts the fatal path.
if (!elector.parentZNodeExists()) { /* expect formatZK or ACL fix */ }

Try / catch

try {
  zkfcMain.run(args); // daemon entry
} catch (RuntimeException e) {
  if (e.getMessage().startsWith("ZK Failover Controller failed:")) {
    // fatal: supervise-restart AFTER fixing the logged root cause
    // ('Fatal error occurred:' line), else the daemon will crash-loop
  }
}

Prevention

Monitor ZooKeeper quorum health and /hadoop-ha/<ns> znode ACLs as part of HA monitoring.
Keep ha.zookeeper.quorum accurate on all nodes; a single stale host is tolerable, a full stale list is fatal.
Run 'hdfs zkfc -formatZK -force' only in controlled windows; wrong ownership of /hadoop-ha is a classic fatal cause.
Supervise ZKFC with restart backoff so crash-loops on a transient ZK outage don't hot-loop.

When it happens

Trigger: ActiveStandbyElector reports a fatal error: cannot connect to any ZooKeeper server in ha.zookeeper.quorum, session establishment keeps failing, or creating the /hadoop-ha/<nameservice> parent znode fails (bad ACLs, permission denied, existing node owned by another user).

Common situations: All ZooKeepers down or wrong ha.zookeeper.quorum/port; SASL/Digest auth to ZK failing; /hadoop-ha znode created by another user or with restrictive ACL; ZK ensemble unreachable due to firewall after a network change.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/44eab2af487a81a3. Report an issue: GitHub.