{"record":{"id":"f04401e59df29755","repo":"apache/hadoop","slug":"unable-to-failover-to","errorCode":null,"errorMessage":"Unable to failover to {}","messagePattern":"Unable to failover to (.+?)","errorType":"exception","errorClass":"FailoverFailedException","httpStatus":null,"severity":"critical","filePath":"hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/FailoverController.java","lineNumber":258,"sourceCode":"\n    // We failed to make toSvc active\n    if (failed) {\n      String msg = \"Unable to failover to \" + toSvc;\n      // Only try to failback if we didn't fence fromSvc\n      if (!tryFence) {\n        try {\n          // Unconditionally fence toSvc in case it is still trying to\n          // become active, eg we timed out waiting for its response.\n          // Unconditionally force fromSvc to become active since it\n          // was previously active when we initiated failover.\n          failover(toSvc, fromSvc, true, true);\n        } catch (FailoverFailedException ffe) {\n          msg += \". Failback to \" + fromSvc +\n            \" failed (\" + ffe.getMessage() + \")\";\n          LOG.error(msg);\n        }\n      }\n      throw new FailoverFailedException(msg, cause);\n    }\n  }\n}\n","sourceCodeStart":240,"sourceCodeEnd":262,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/FailoverController.java#L240-L262","documentation":"FailoverController.failover() throws FailoverFailedException(msg='Unable to failover to <target>', cause) after transitionToActive on the target failed AND the unconditional fail-back attempt (failover(toSvc, fromSvc, true, true)) also failed. The cluster may be left with no active node, so this is the worst-case failover outcome.","triggerScenarios":"transitionToActive(toSvc) threw (e.g. the new active NameNode cannot start because JournalNodes/QJM are unavailable, storage not formatted, upgrade in progress) and the automatic rollback that tries to make the original node active again also threw FailoverFailedException, so the original message gets '. Failback to X failed (...)' appended and is rethrown with the original cause.","commonSituations":"Shared edits (QJM) unavailable so neither node can become active; both nodes unhealthy; JournalNodes down or torn during the failover; forced failover where the old active is unreachable and fencing fails during fail-back.","solutions":["Read the nested cause chain and the NameNode log of the intended active — the root ServiceFailedException/Throwable says why transitionToActive refused (usually JournalNode/shared-edits connectivity).","Restore the shared edits infrastructure (all JournalNodes up, dfs.namenode.shared.edits.dir correct) before retrying.","Check whether any node is active ('hdfs haadmin -getAllServiceState') and if none, manually recover: 'hdfs haadmin -transitionToActive <nnId>' after fixing the blocker.","If fail-back failed due to fencing, fix the fencing configuration first (see 'Unable to fence' handling)."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"// Before failover, confirm the target can reach the shared edits (JournalNodes)\n// e.g. from the target node: verify QJM connectivity\n// hdfs --daemon list journalnode / nc each dfs.journalnode.edits.dir address\n// Also pre-check both services' health via HAServiceProtocol.monitorHealth().","typeGuard":null,"tryCatchPattern":"try {\n  fc.failover(fromSvc, toSvc, forceActive, forceFence);\n} catch (FailoverFailedException ffe) {\n  // Failback may also have failed: check whether ANY node is active\n  // hdfs haadmin -getAllServiceState\n  // Page on-call; manual recovery path: fix JournalNodes, then\n  // hdfs haadmin -transitionToActive <healthy-nn> after verification.\n  log.error(\"Failover failed and may have left no active: {}\", ffe.getMessage(), ffe.getCause());\n}","preventionTips":["Keep all JournalNodes monitored and sized (an odd, majority-available set) before any failover window.","Never run forced failover when the shared edits layer is degraded.","Automate failover via ZKFC election, which retries safely, instead of manual FailoverController.failover with force flags."],"tags":["hadoop","high-availability","failover","name-node","qjm"],"backgroundTag":"high-availability-failover-failed","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}