apache/hadoop · error · ServiceFailedException

{} is in observer state. Cannot be failover target

Error message

{} is in observer state. Cannot be failover target

What it means

Thrown by ZKFailoverController.checkEligibleForFailover() when a graceful failover is requested to this ZKFC's node while the local service state is HAServiceState.OBSERVER (an HDFS Observer NameNode). The health check runs first and passes, but observer nodes are deliberately excluded from becoming active, so the operation is refused with ServiceFailedException. This protects against promoting a read-only observer into the writer role.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/ZKFailoverController.java:775

    oldZkfc.cedeActive(timeout);
    return oldZkfc;
  }

  /**
   * If the local node is an observer or is unhealthy it
   * is not eligible for graceful failover.
   * @throws ServiceFailedException if the node is an observer or unhealthy
   */
  private synchronized void checkEligibleForFailover()
      throws ServiceFailedException {
    // Check health
    if (this.getLastHealthState() != State.SERVICE_HEALTHY) {
      throw new ServiceFailedException(
          localTarget + " is not currently healthy. " +
          "Cannot be failover target");
    }
    if (serviceState == HAServiceState.OBSERVER) {
      throw new ServiceFailedException(
          localTarget + " is in observer state. " +
          "Cannot be failover target");
    }
  }

  /**
   * @return an {@link HAServiceTarget} for the current active node
   * in the cluster, or null if no node is active.
   * @throws IOException if a ZK-related issue occurs
   * @throws InterruptedException if thread is interrupted 
   */
  private HAServiceTarget getCurrentActive()
      throws IOException, InterruptedException {
    synchronized (elector) {
      synchronized (this) {
        byte[] activeData;
        try {
          activeData = elector.getActiveData();

View on GitHub (pinned to 2add963021)

Solutions

1. Pick a different target: run 'hdfs haadmin -getAllServiceState' and fail over to a node whose state is standby, not observer
2. If the observer node must become failover-capable, reconfigure and restart that NameNode as a standby (remove dfs.ha.nn.observer.<id>=true) so its ZKFC can elect it
3. Verify you are not hitting the sibling health failure: the message 'is not currently healthy' is checked before the observer check

Example fix

# before
hdfs haadmin -failover nn1 nn2   # nn2 is an ObserverNode -> ServiceFailedException

# after
hdfs haadmin -getAllServiceState    # confirm states first
hdfs haadmin -failover nn1 nn3      # target a standby node

HAServiceProtocol proxy = ...; // to the target NN
HAServiceStatus status = proxy.getServiceStatus();
if (status.getState() == HAServiceState.OBSERVER) {
  throw new IllegalStateException("target is an observer; pick a standby node");
}
if (status.getState() != HAServiceState.STANDBY) {
  throw new IllegalStateException("target must be standby, was " + status.getState());
}

try {
  zfc.gracefulFailover(0); // ZKFCProtocol on target
} catch (ServiceFailedException e) {
  // message distinguishes 'not currently healthy' vs 'is in observer state'
  LOG.warn("graceful failover refused: {}", e.getMessage());
}

When it happens

Trigger: Running 'hdfs haadmin -failover <from> <to>' (or the ZKFCProtocol.gracefulFailover RPC on port 8019) where the ZKFC on <to> is managing a NameNode started with dfs.ha.nn.observer.<id>=true; also failover scripts that iterate over all NNs and pick the first healthy one, hitting the observer.

Common situations: Observer NameNode deployments (HDFS-12943) where an operator or automation tool targets the observer for failover; test clusters where nodes were converted to observers after the failover script was written.

Related errors

• Unexpected ZooKeeper issue fetching active node info
• Cannot transition from 'observer' to 'active'
• Cannot transition from 'active' to 'observer'
• Request from ZK failover controller at {Server.getRemoteAddr
• No fencer configured for {}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/da2ae81b2726db2b. Report an issue: GitHub.