{"record":{"id":"44eab2af487a81a3","repo":"apache/hadoop","slug":"zk-failover-controller-failed","errorCode":null,"errorMessage":"ZK Failover Controller failed: {}","messagePattern":"ZK Failover Controller failed: (.+?)","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"critical","filePath":"hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/ZKFailoverController.java","lineNumber":401,"sourceCode":"        zkTimeout, getParentZnode(), zkAcls, zkAuths,\n        new ElectorCallbacks(), maxRetryNum, truststoreKeystore);\n  }\n  \n  private String getParentZnode() {\n    String znode = conf.get(ZK_PARENT_ZNODE_KEY,\n        ZK_PARENT_ZNODE_DEFAULT);\n    if (!znode.endsWith(\"/\")) {\n      znode += \"/\";\n    }\n    return znode + getScopeInsideParentNode();\n  }\n\n  private synchronized void mainLoop() throws InterruptedException {\n    while (fatalError == null) {\n      wait();\n    }\n    assert fatalError != null; // only get here on fatal\n    throw new RuntimeException(\n        \"ZK Failover Controller failed: \" + fatalError);\n  }\n  \n  private synchronized void fatalError(String err) {\n    LOG.error(\"Fatal error occurred:\" + err);\n    fatalError = err;\n    notifyAll();\n  }\n  \n  private synchronized void becomeActive() throws ServiceFailedException {\n    LOG.info(\"Trying to make \" + localTarget + \" active...\");\n    try {\n      HAServiceProtocolHelper.transitionToActive(localTarget.getProxy(\n          conf, FailoverController.getRpcTimeoutToNewActive(conf)),\n          createReqInfo());\n      String msg = \"Successfully transitioned \" + localTarget +\n          \" to active state\";\n      LOG.info(msg);","sourceCodeStart":383,"sourceCodeEnd":419,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/ZKFailoverController.java#L383-L419","documentation":"ZKFailoverController.mainLoop() blocks until fatalError is set by fatalError(String) — called on unrecoverable conditions such as fatal ZooKeeper events or parent-znode creation failure — then throws RuntimeException('ZK Failover Controller failed: <err>'). It means the ZKFC process hit a condition it cannot recover from and is exiting.","triggerScenarios":"ActiveStandbyElector reports a fatal error: cannot connect to any ZooKeeper server in ha.zookeeper.quorum, session establishment keeps failing, or creating the /hadoop-ha/<nameservice> parent znode fails (bad ACLs, permission denied, existing node owned by another user).","commonSituations":"All ZooKeepers down or wrong ha.zookeeper.quorum/port; SASL/Digest auth to ZK failing; /hadoop-ha znode created by another user or with restrictive ACL; ZK ensemble unreachable due to firewall after a network change.","solutions":["Read the ZKFC log line 'Fatal error occurred:<err>' immediately before the exception — it names the exact condition.","Verify ZooKeeper health: zkCli.sh (or 'echo ruok | nc zk 2181') against every host in ha.zookeeper.quorum and check client port reachability.","Fix znode permissions/ownership, or re-create the parent znode with 'hdfs zkfc -formatZK -force' (destroys HA state — use with caution) after fixing ACLs/auth.","Restart the ZKFC daemon once the underlying cause is fixed."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"// Before starting / during ops: verify the ZK quorum and parent znode\n// from the ZKFC host:\n//   zkCli.sh -server zk1:2181 ls /hadoop-ha\n// Any ConnectTimeout/NoNode/ACL failure here predicts the fatal path.\nif (!elector.parentZNodeExists()) { /* expect formatZK or ACL fix */ }","typeGuard":null,"tryCatchPattern":"try {\n  zkfcMain.run(args); // daemon entry\n} catch (RuntimeException e) {\n  if (e.getMessage().startsWith(\"ZK Failover Controller failed:\")) {\n    // fatal: supervise-restart AFTER fixing the logged root cause\n    // ('Fatal error occurred:' line), else the daemon will crash-loop\n  }\n}","preventionTips":["Monitor ZooKeeper quorum health and /hadoop-ha/<ns> znode ACLs as part of HA monitoring.","Keep ha.zookeeper.quorum accurate on all nodes; a single stale host is tolerable, a full stale list is fatal.","Run 'hdfs zkfc -formatZK -force' only in controlled windows; wrong ownership of /hadoop-ha is a classic fatal cause.","Supervise ZKFC with restart backoff so crash-loops on a transient ZK outage don't hot-loop."],"tags":["hadoop","high-availability","zkfc","zookeeper","fatal"],"backgroundTag":"zookeeper-connection-failed","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}