apache/hadoop · critical · FailoverFailedException

Unable to fence {}. Fencing failed.

Error message

Unable to fence {}. Fencing failed.

What it means

FailoverController throws FailoverFailedException('Unable to fence <target>. Fencing failed.') when fromSvc.getFencer().fence(fromSvc, toSvc) returns false — the configured fencing method(s) could not guarantee the old active is cut off. Failover stops because proceeding could leave two active nodes (split-brain).

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/FailoverController.java:217

                       HAServiceTarget toSvc,
                       boolean forceFence,
                       boolean forceActive)
      throws FailoverFailedException {
    Preconditions.checkArgument(fromSvc.getFencer() != null,
        "failover requires a fencer");
    preFailoverChecks(fromSvc, toSvc, forceActive);

    // Try to make fromSvc standby
    boolean tryFence = true;
    
    if (tryGracefulFence(fromSvc)) {
      tryFence = forceFence;
    }

    // Fence fromSvc if it's required or forced by the user
    if (tryFence) {
      if (!fromSvc.getFencer().fence(fromSvc, toSvc)) {
        throw new FailoverFailedException("Unable to fence " +
            fromSvc + ". Fencing failed.");
      }
    }

    // Try to make toSvc active
    boolean failed = false;
    Throwable cause = null;
    try {
      HAServiceProtocolHelper.transitionToActive(
          toSvc.getProxy(conf, rpcTimeoutToNewActive),
          createReqInfo());
    } catch (ServiceFailedException sfe) {
      LOG.error("Unable to make {} active ({}). Failing back.",
          toSvc, sfe.getMessage());
      failed = true;
      cause = sfe;
    } catch (IOException ioe) {
      LOG.error("Unable to make {} active (unable to connect). Failing back.",

View on GitHub (pinned to 2add963021)

Solutions

1. Run the fencing command manually as the HDFS user on the standby host (e.g. ssh to the old active with the configured key/user/port) to see the real failure.
2. For sshfence: set up passwordless SSH (ssh-keygen/authorized_keys) and configure 'sshfence(<user>:<port>)' plus dfs.ha.fencing.ssh.private-key-files if needed.
3. For shell: fix the script so it exits 0 exactly when fencing succeeded; ShellCommandFencer logs the command and exit code.
4. Fix network/DNS to the old active host, then retry the failover.

Example fix

<!-- before: sshfence without working SSH setup -->
<property><name>dfs.ha.fencing.methods</name><value>sshfence</value></property>

<!-- after: explicit user and port, key files configured -->
<property><name>dfs.ha.fencing.methods</name><value>sshfence(hdfs:22)</value></property>
<property><name>dfs.ha.fencing.ssh.private-key-files</name><value>/home/hdfs/.ssh/id_rsa</value></property>

// Fail fast on missing fencing configuration before failover
try {
  fromSvc.checkFencingConfigured(); // throws BadFencingConfigurationException if unset
} catch (BadFencingConfigurationException e) {
  // fix dfs.ha.fencing.methods before any failover can be attempted
}

// Optionally smoke-test a shell fencer with a harmless command first
new NodeFencer(conf, "shell(/bin/true)"); // exercises config parsing only

try {
  fc.failover(fromSvc, toSvc, false, forceFence);
} catch (FailoverFailedException ffe) {
  if (ffe.getMessage().startsWith("Unable to fence")) {
    // fencing failed: treat as split-brain risk, alert immediately,
    // do NOT auto-promote the target; verify old active state manually
  }
}

When it happens

Trigger: failover() with forceFence=true, or graceful demotion of the old active failed (tryGracefulFence returned false), and then the fencer failed: sshfence could not SSH to the old active and kill its process, or a shell(...) fencer script exited non-zero.

Common situations: sshfence: no passwordless SSH key from standby to active, wrong SSH user/port, sshd not reachable on the configured port, or dfs.ha.fencing.ssh.private-key-files not set; shell fencer: script missing, not executable, bug in script, or returns non-zero; fencing config valid but the network path between nodes is broken.

Related errors

• Unable to fence {}
• No argument passed to 'shell' fencing method
• Expecting arguments size of at most two, getting {}
• Unable to parse user and SSH port: {}
• Port number '{}' invalid

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/162d5b44481534a3. Report an issue: GitHub.