{"record":{"id":"975f2304e8616b60","repo":"apache/hadoop","slug":"unable-to-fence","errorCode":null,"errorMessage":"Unable to fence {}","messagePattern":"Unable to fence (.+?)","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"critical","filePath":"hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/ZKFailoverController.java","lineNumber":567,"sourceCode":"        RequestSource.REQUEST_BY_ZKFC).tryGracefulFence(target);\n    if (gracefulWorked) {\n      // It's possible that it's in standby but just about to go into active,\n      // no? Is there some race here?\n      LOG.info(\"Successfully transitioned \" + target + \" to standby \" +\n          \"state without fencing\");\n      return;\n    }\n    \n    try {\n      target.checkFencingConfigured();\n    } catch (BadFencingConfigurationException e) {\n      LOG.error(\"Couldn't fence old active \" + target, e);\n      recordActiveAttempt(new ActiveAttemptRecord(false, \"Unable to fence old active\"));\n      throw new RuntimeException(e);\n    }\n    \n    if (!target.getFencer().fence(target)) {\n      throw new RuntimeException(\"Unable to fence \" + target);\n    }\n  }\n\n\n  /**\n   * Request from graceful failover to cede active role. Causes\n   * this ZKFC to transition its local node to standby, then quit\n   * the election for the specified period of time, after which it\n   * will rejoin iff it is healthy.\n   */\n  void cedeActive(final int millisToCede)\n      throws AccessControlException, ServiceFailedException, IOException {\n    try {\n      UserGroupInformation.getLoginUser().doAs(new PrivilegedExceptionAction<Void>() {\n        @Override\n        public Void run() throws Exception {\n          doCedeActive(millisToCede);\n          return null;","sourceCodeStart":549,"sourceCodeEnd":585,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/ZKFailoverController.java#L549-L585","documentation":"During graceful failover, ZKFailoverController.fenceOldActive() throws RuntimeException('Unable to fence <target>') when target.getFencer().fence(target) returns false — every configured fencing method failed to fence the old active, so the local node must not become active (split-brain risk). A sibling path also throws if BadFencingConfigurationException occurs (fencing not configured).","triggerScenarios":"Graceful failover ('hdfs haadmin -failover' / gracefulFailover) where the old active did not yield via cedeActive in time and the fallback fencing failed: sshfence could not establish the SSH session and kill the process, or the shell fencer's command exited non-zero.","commonSituations":"Passwordless SSH from the new active to the old active not set up; sshd unreachable on the old active while its NameNode still runs; broken fencing script; fencing config lines malformed (the sibling BadFencingConfigurationException path).","solutions":["Test the fencing method manually from the node that would become active (SSH with the configured key/user/port to the old active, or run the shell script and check its exit code).","Fix sshfence/shell setup: keys, dfs.ha.fencing.ssh.private-key-files, correct 'sshfence(user:port)', executable script returning 0 on success.","Check network reachability between the two HA nodes, then retry the failover.","If fencing is legitimately impossible (old active host destroyed), verify the old NameNode is truly dead on all peers before manually forcing the transition."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"// Before graceful failover: confirm fencing is configured and testable\ntarget.checkFencingConfigured(); // BadFencingConfigurationException if not\n// And confirm SSH path works from this host to the old active:\n//   ssh -o BatchMode=yes -p 22 <user>@<oldActive> true\n// exit code 0 predicts sshfence success","typeGuard":null,"tryCatchPattern":"try {\n  zkfc.gracefulFailover(); // via HAAdmin / ZKFCProtocol\n} catch (RuntimeException e) {\n  if (e.getMessage() != null && e.getMessage().startsWith(\"Unable to fence\")) {\n    // split-brain risk: verify the old active is truly dead or fenced\n    // before any manual promotion; page on-call\n  }\n}","preventionTips":["Validate fencing (SSH keys, script exit codes) from every node during HA drills.","Include a scheduled failover test in staging so fencing breakage is caught before production incidents.","Never assume 'old active unreachable' means 'old active dead' — that assumption causes split-brain."],"tags":["hadoop","high-availability","zkfc","fencing","split-brain","graceful-failover"],"backgroundTag":"node-fencing-failed","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}