apache/hadoop · critical · FailoverFailedException

Unable to failover to {}

Error message

Unable to failover to {}

What it means

FailoverController.failover() throws FailoverFailedException(msg='Unable to failover to <target>', cause) after transitionToActive on the target failed AND the unconditional fail-back attempt (failover(toSvc, fromSvc, true, true)) also failed. The cluster may be left with no active node, so this is the worst-case failover outcome.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/FailoverController.java:258

    // We failed to make toSvc active
    if (failed) {
      String msg = "Unable to failover to " + toSvc;
      // Only try to failback if we didn't fence fromSvc
      if (!tryFence) {
        try {
          // Unconditionally fence toSvc in case it is still trying to
          // become active, eg we timed out waiting for its response.
          // Unconditionally force fromSvc to become active since it
          // was previously active when we initiated failover.
          failover(toSvc, fromSvc, true, true);
        } catch (FailoverFailedException ffe) {
          msg += ". Failback to " + fromSvc +
            " failed (" + ffe.getMessage() + ")";
          LOG.error(msg);
        }
      }
      throw new FailoverFailedException(msg, cause);
    }
  }
}

View on GitHub (pinned to 2add963021)

Solutions

1. Read the nested cause chain and the NameNode log of the intended active — the root ServiceFailedException/Throwable says why transitionToActive refused (usually JournalNode/shared-edits connectivity).
2. Restore the shared edits infrastructure (all JournalNodes up, dfs.namenode.shared.edits.dir correct) before retrying.
3. Check whether any node is active ('hdfs haadmin -getAllServiceState') and if none, manually recover: 'hdfs haadmin -transitionToActive <nnId>' after fixing the blocker.
4. If fail-back failed due to fencing, fix the fencing configuration first (see 'Unable to fence' handling).

// Before failover, confirm the target can reach the shared edits (JournalNodes)
// e.g. from the target node: verify QJM connectivity
// hdfs --daemon list journalnode / nc each dfs.journalnode.edits.dir address
// Also pre-check both services' health via HAServiceProtocol.monitorHealth().

try {
  fc.failover(fromSvc, toSvc, forceActive, forceFence);
} catch (FailoverFailedException ffe) {
  // Failback may also have failed: check whether ANY node is active
  // hdfs haadmin -getAllServiceState
  // Page on-call; manual recovery path: fix JournalNodes, then
  // hdfs haadmin -transitionToActive <healthy-nn> after verification.
  log.error("Failover failed and may have left no active: {}", ffe.getMessage(), ffe.getCause());
}

When it happens

Trigger: transitionToActive(toSvc) threw (e.g. the new active NameNode cannot start because JournalNodes/QJM are unavailable, storage not formatted, upgrade in progress) and the automatic rollback that tries to make the original node active again also threw FailoverFailedException, so the original message gets '. Failback to X failed (...)' appended and is rethrown with the original cause.

Common situations: Shared edits (QJM) unavailable so neither node can become active; both nodes unhealthy; JournalNodes down or torn during the failover; forced failover where the old active is unreachable and fencing fails during fail-back.

Related errors

• Got an IO exception
• Couldn't transition to active
• Transition from state {} to {} is not allowed.
• ${className} does not support seekToNewSource.
• Unable to fence {}. Fencing failed.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/f04401e59df29755. Report an issue: GitHub.