apache/hadoop · error · ServiceFailedException

{} is not currently healthy. Cannot be failover target

Error message

{} is not currently healthy. Cannot be failover target

What it means

checkEligibleForFailover() throws ServiceFailedException('<target> is not currently healthy. Cannot be failover target') when graceful failover is requested and the local node's last health-monitor state is not SERVICE_HEALTHY (a sibling check rejects observer-state nodes). The pre-flight gate stops failover onto a node that cannot safely take over.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/ZKFailoverController.java:770

  private ZKFCProtocol cedeRemoteActive(HAServiceTarget remote, int timeout)
    throws IOException {
    LOG.info("Asking " + remote + " to cede its active state for "
               + timeout + "ms");
    ZKFCProtocol oldZkfc = remote.getZKFCProxy(conf, timeout);
    oldZkfc.cedeActive(timeout);
    return oldZkfc;
  }

  /**
   * If the local node is an observer or is unhealthy it
   * is not eligible for graceful failover.
   * @throws ServiceFailedException if the node is an observer or unhealthy
   */
  private synchronized void checkEligibleForFailover()
      throws ServiceFailedException {
    // Check health
    if (this.getLastHealthState() != State.SERVICE_HEALTHY) {
      throw new ServiceFailedException(
          localTarget + " is not currently healthy. " +
          "Cannot be failover target");
    }
    if (serviceState == HAServiceState.OBSERVER) {
      throw new ServiceFailedException(
          localTarget + " is in observer state. " +
          "Cannot be failover target");
    }
  }

  /**
   * @return an {@link HAServiceTarget} for the current active node
   * in the cluster, or null if no node is active.
   * @throws IOException if a ZK-related issue occurs
   * @throws InterruptedException if thread is interrupted 
   */
  private HAServiceTarget getCurrentActive()
      throws IOException, InterruptedException {

View on GitHub (pinned to 2add963021)

Solutions

1. Check the target's health: 'hdfs haadmin -checkHealth <nnId>' and the target NameNode's logs; fix whatever the check reports.
2. Wait for the ZKFC health monitor to report SERVICE_HEALTHY (first check completes shortly after start) before issuing failover.
3. Make sure the failover target is a standby, not an observer node.
4. Retry the failover once health is green.

// Same gate the ZKFC applies, run before issuing the failover
zkfc.checkEligibleForFailover(); // ServiceFailedException if not SERVICE_HEALTHY or observer
// external equivalent: hdfs haadmin -checkHealth <nnId>

try {
  zkfc.gracefulFailover();
} catch (ServiceFailedException e) {
  if (e.getMessage().endsWith("Cannot be failover target")) {
    // target unhealthy or observer: check NN logs/health, wait for
    // SERVICE_HEALTHY (first health check after ZKFC start), then retry
  }
}

When it happens

Trigger: Running 'hdfs haadmin -failover' (gracefulFailover) where the intended new active's health monitor last reported anything other than healthy: a failed checkHealth, an unreachable local service, or the monitor not having completed its first check yet after ZKFC startup.

Common situations: Standby NameNode down or unhealthy (shared edits dir missing, disk issues, OOM); failover issued seconds after ZKFC start before the first health check completes; target actually an observer node (sibling 'observer state' message); transient health-check RPC failures.

Related errors

• Unable to become active. Service became unhealthy while tryi
• Unable to fence {}
• No other node is currently active.
• Unable to become active. Local node did not get an opportuni
• Failed to become active. {}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/ff658307aa3639e6. Report an issue: GitHub.