apache/hadoop · critical · FailoverFailedException

Unable to fence . Fencing failed.

Error message

Unable to fence {}. Fencing failed.

What it means

FailoverController throws FailoverFailedException('Unable to fence <target>. Fencing failed.') when fromSvc.getFencer().fence(fromSvc, toSvc) returns false — the configured fencing method(s) could not guarantee the old active is cut off. Failover stops because proceeding could leave two active nodes (split-brain).

Solutions

1. Run the fencing command manually as the HDFS user on the standby host (e.g. ssh to the old active with the configured key/user/port) to see the real failure.
2. For sshfence: set up passwordless SSH (ssh-keygen/authorized_keys) and configure 'sshfence(<user>:<port>)' plus dfs.ha.fencing.ssh.private-key-files if needed.
3. For shell: fix the script so it exits 0 exactly when fencing succeeded; ShellCommandFencer logs the command and exit code.
4. Fix network/DNS to the old active host, then retry the failover.

Example fix

<!-- before: sshfence without working SSH setup -->
<property><name>dfs.ha.fencing.methods</name><value>sshfence</value></property>

<!-- after: explicit user and port, key files configured -->
<property><name>dfs.ha.fencing.methods</name><value>sshfence(hdfs:22)</value></property>
<property><name>dfs.ha.fencing.ssh.private-key-files</name><value>/home/hdfs/.ssh/id_rsa</value></property>

// Fail fast on missing fencing configuration before failover
try {
  fromSvc.checkFencingConfigured(); // throws BadFencingConfigurationException if unset
} catch (BadFencingConfigurationException e) {
  // fix dfs.ha.fencing.methods before any failover can be attempted
}

// Optionally smoke-test a shell fencer with a harmless command first
new NodeFencer(conf, "shell(/bin/true)"); // exercises config parsing only

try {
  fc.failover(fromSvc, toSvc, false, forceFence);
} catch (FailoverFailedException ffe) {
  if (ffe.getMessage().startsWith("Unable to fence")) {
    // fencing failed: treat as split-brain risk, alert immediately,
    // do NOT auto-promote the target; verify old active state manually
  }
}

When it happens

Trigger: failover() with forceFence=true, or graceful demotion of the old active failed (tryGracefulFence returned false), and then the fencer failed: sshfence could not SSH to the old active and kill its process, or a shell(...) fencer script exited non-zero.

Common situations: sshfence: no passwordless SSH key from standby to active, wrong SSH user/port, sshd not reachable on the configured port, or dfs.ha.fencing.ssh.private-key-files not set; shell fencer: script missing, not executable, bug in script, or returns non-zero; fencing config valid but the network path between nodes is broken.

Related errors

• Unable to fence
• Expecting arguments size of at most two, getting
• No argument passed to 'shell' fencing method
• Port number ' ' invalid
• Unable to parse user and SSH port

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/162d5b44481534a3. Report an issue: GitHub.

Appendix: source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/FailoverController.java:217

                       HAServiceTarget toSvc,
                       boolean forceFence,
                       boolean forceActive)
      throws FailoverFailedException {
    Preconditions.checkArgument(fromSvc.getFencer() != null,
        "failover requires a fencer");
    preFailoverChecks(fromSvc, toSvc, forceActive);

    // Try to make fromSvc standby
    boolean tryFence = true;
    
    if (tryGracefulFence(fromSvc)) {
      tryFence = forceFence;
    }

    // Fence fromSvc if it's required or forced by the user
    if (tryFence) {
      if (!fromSvc.getFencer().fence(fromSvc, toSvc)) {
        throw new FailoverFailedException("Unable to fence " +
            fromSvc + ". Fencing failed.");
      }
    }

    // Try to make toSvc active
    boolean failed = false;
    Throwable cause = null;
    try {
      HAServiceProtocolHelper.transitionToActive(
          toSvc.getProxy(conf, rpcTimeoutToNewActive),
          createReqInfo());
    } catch (ServiceFailedException sfe) {
      LOG.error("Unable to make {} active ({}). Failing back.",
          toSvc, sfe.getMessage());
      failed = true;
      cause = sfe;
    } catch (IOException ioe) {
      LOG.error("Unable to make {} active (unable to connect). Failing back.",

View on GitHub (pinned to 2add963021)