{"record":{"id":"162d5b44481534a3","repo":"apache/hadoop","slug":"unable-to-fence-fencing-failed","errorCode":null,"errorMessage":"Unable to fence {}. Fencing failed.","messagePattern":"Unable to fence (.+?)\\. Fencing failed\\.","errorType":"exception","errorClass":"FailoverFailedException","httpStatus":null,"severity":"critical","filePath":"hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/FailoverController.java","lineNumber":217,"sourceCode":"                       HAServiceTarget toSvc,\n                       boolean forceFence,\n                       boolean forceActive)\n      throws FailoverFailedException {\n    Preconditions.checkArgument(fromSvc.getFencer() != null,\n        \"failover requires a fencer\");\n    preFailoverChecks(fromSvc, toSvc, forceActive);\n\n    // Try to make fromSvc standby\n    boolean tryFence = true;\n    \n    if (tryGracefulFence(fromSvc)) {\n      tryFence = forceFence;\n    }\n\n    // Fence fromSvc if it's required or forced by the user\n    if (tryFence) {\n      if (!fromSvc.getFencer().fence(fromSvc, toSvc)) {\n        throw new FailoverFailedException(\"Unable to fence \" +\n            fromSvc + \". Fencing failed.\");\n      }\n    }\n\n    // Try to make toSvc active\n    boolean failed = false;\n    Throwable cause = null;\n    try {\n      HAServiceProtocolHelper.transitionToActive(\n          toSvc.getProxy(conf, rpcTimeoutToNewActive),\n          createReqInfo());\n    } catch (ServiceFailedException sfe) {\n      LOG.error(\"Unable to make {} active ({}). Failing back.\",\n          toSvc, sfe.getMessage());\n      failed = true;\n      cause = sfe;\n    } catch (IOException ioe) {\n      LOG.error(\"Unable to make {} active (unable to connect). Failing back.\",","sourceCodeStart":199,"sourceCodeEnd":235,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/FailoverController.java#L199-L235","documentation":"FailoverController throws FailoverFailedException('Unable to fence <target>. Fencing failed.') when fromSvc.getFencer().fence(fromSvc, toSvc) returns false — the configured fencing method(s) could not guarantee the old active is cut off. Failover stops because proceeding could leave two active nodes (split-brain).","triggerScenarios":"failover() with forceFence=true, or graceful demotion of the old active failed (tryGracefulFence returned false), and then the fencer failed: sshfence could not SSH to the old active and kill its process, or a shell(...) fencer script exited non-zero.","commonSituations":"sshfence: no passwordless SSH key from standby to active, wrong SSH user/port, sshd not reachable on the configured port, or dfs.ha.fencing.ssh.private-key-files not set; shell fencer: script missing, not executable, bug in script, or returns non-zero; fencing config valid but the network path between nodes is broken.","solutions":["Run the fencing command manually as the HDFS user on the standby host (e.g. ssh to the old active with the configured key/user/port) to see the real failure.","For sshfence: set up passwordless SSH (ssh-keygen/authorized_keys) and configure 'sshfence(<user>:<port>)' plus dfs.ha.fencing.ssh.private-key-files if needed.","For shell: fix the script so it exits 0 exactly when fencing succeeded; ShellCommandFencer logs the command and exit code.","Fix network/DNS to the old active host, then retry the failover."],"exampleFix":"<!-- before: sshfence without working SSH setup -->\n<property><name>dfs.ha.fencing.methods</name><value>sshfence</value></property>\n\n<!-- after: explicit user and port, key files configured -->\n<property><name>dfs.ha.fencing.methods</name><value>sshfence(hdfs:22)</value></property>\n<property><name>dfs.ha.fencing.ssh.private-key-files</name><value>/home/hdfs/.ssh/id_rsa</value></property>","handlingStrategy":"validation","validationCode":"// Fail fast on missing fencing configuration before failover\ntry {\n  fromSvc.checkFencingConfigured(); // throws BadFencingConfigurationException if unset\n} catch (BadFencingConfigurationException e) {\n  // fix dfs.ha.fencing.methods before any failover can be attempted\n}\n\n// Optionally smoke-test a shell fencer with a harmless command first\nnew NodeFencer(conf, \"shell(/bin/true)\"); // exercises config parsing only","typeGuard":null,"tryCatchPattern":"try {\n  fc.failover(fromSvc, toSvc, false, forceFence);\n} catch (FailoverFailedException ffe) {\n  if (ffe.getMessage().startsWith(\"Unable to fence\")) {\n    // fencing failed: treat as split-brain risk, alert immediately,\n    // do NOT auto-promote the target; verify old active state manually\n  }\n}","preventionTips":["During HA setup, test fencing from every node: actually run the sshfence SSH login and the shell script against a sacrificial host.","Keep fencing methods ordered with the most reliable first; ensure the chosen method can succeed when the old active host is up but unresponsive.","Add dfs.ha.fencing.ssh.connect-timeout and private-key config checks to cluster validation scripts."],"tags":["hadoop","high-availability","fencing","ssh","shell","split-brain"],"backgroundTag":"node-fencing-failed","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}