What causes "Unable to fence {}" in apache/hadoop?

Test the fencing method manually from the node that would become active (SSH with the configured key/user/port to the old active, or run the shell script and check its exit code).

apache/hadoop · critical · RuntimeException

Unable to fence {}

Error message

Unable to fence {}

What it means

During graceful failover, ZKFailoverController.fenceOldActive() throws RuntimeException('Unable to fence <target>') when target.getFencer().fence(target) returns false — every configured fencing method failed to fence the old active, so the local node must not become active (split-brain risk). A sibling path also throws if BadFencingConfigurationException occurs (fencing not configured).

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/ZKFailoverController.java:567

        RequestSource.REQUEST_BY_ZKFC).tryGracefulFence(target);
    if (gracefulWorked) {
      // It's possible that it's in standby but just about to go into active,
      // no? Is there some race here?
      LOG.info("Successfully transitioned " + target + " to standby " +
          "state without fencing");
      return;
    }
    
    try {
      target.checkFencingConfigured();
    } catch (BadFencingConfigurationException e) {
      LOG.error("Couldn't fence old active " + target, e);
      recordActiveAttempt(new ActiveAttemptRecord(false, "Unable to fence old active"));
      throw new RuntimeException(e);
    }
    
    if (!target.getFencer().fence(target)) {
      throw new RuntimeException("Unable to fence " + target);
    }
  }


  /**
   * Request from graceful failover to cede active role. Causes
   * this ZKFC to transition its local node to standby, then quit
   * the election for the specified period of time, after which it
   * will rejoin iff it is healthy.
   */
  void cedeActive(final int millisToCede)
      throws AccessControlException, ServiceFailedException, IOException {
    try {
      UserGroupInformation.getLoginUser().doAs(new PrivilegedExceptionAction<Void>() {
        @Override
        public Void run() throws Exception {
          doCedeActive(millisToCede);
          return null;

View on GitHub (pinned to 2add963021)

Solutions

Test the fencing method manually from the node that would become active (SSH with the configured key/user/port to the old active, or run the shell script and check its exit code).
Fix sshfence/shell setup: keys, dfs.ha.fencing.ssh.private-key-files, correct 'sshfence(user:port)', executable script returning 0 on success.
Check network reachability between the two HA nodes, then retry the failover.
If fencing is legitimately impossible (old active host destroyed), verify the old NameNode is truly dead on all peers before manually forcing the transition.

Defensive patterns

Strategy: try-catch

Validate before calling

// Before graceful failover: confirm fencing is configured and testable
target.checkFencingConfigured(); // BadFencingConfigurationException if not
// And confirm SSH path works from this host to the old active:
//   ssh -o BatchMode=yes -p 22 <user>@<oldActive> true
// exit code 0 predicts sshfence success

Try / catch

try {
  zkfc.gracefulFailover(); // via HAAdmin / ZKFCProtocol
} catch (RuntimeException e) {
  if (e.getMessage() != null && e.getMessage().startsWith("Unable to fence")) {
    // split-brain risk: verify the old active is truly dead or fenced
    // before any manual promotion; page on-call
  }
}

Prevention

Validate fencing (SSH keys, script exit codes) from every node during HA drills.
Include a scheduled failover test in staging so fencing breakage is caught before production incidents.
Never assume 'old active unreachable' means 'old active dead' — that assumption causes split-brain.

When it happens

Trigger: Graceful failover ('hdfs haadmin -failover' / gracefulFailover) where the old active did not yield via cedeActive in time and the fallback fencing failed: sshfence could not establish the SSH session and kill the process, or the shell fencer's command exited non-zero.

Common situations: Passwordless SSH from the new active to the old active not set up; sshd unreachable on the old active while its NameNode still runs; broken fencing script; fencing config lines malformed (the sibling BadFencingConfigurationException path).

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/975f2304e8616b60. Report an issue: GitHub.