{"record":{"id":"da2ae81b2726db2b","repo":"apache/hadoop","slug":"is-in-observer-state-cannot-be-failover-target","errorCode":null,"errorMessage":"{} is in observer state. Cannot be failover target","messagePattern":"(.+?) is in observer state\\. Cannot be failover target","errorType":"exception","errorClass":"ServiceFailedException","httpStatus":null,"severity":"error","filePath":"hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/ZKFailoverController.java","lineNumber":775,"sourceCode":"    oldZkfc.cedeActive(timeout);\n    return oldZkfc;\n  }\n\n  /**\n   * If the local node is an observer or is unhealthy it\n   * is not eligible for graceful failover.\n   * @throws ServiceFailedException if the node is an observer or unhealthy\n   */\n  private synchronized void checkEligibleForFailover()\n      throws ServiceFailedException {\n    // Check health\n    if (this.getLastHealthState() != State.SERVICE_HEALTHY) {\n      throw new ServiceFailedException(\n          localTarget + \" is not currently healthy. \" +\n          \"Cannot be failover target\");\n    }\n    if (serviceState == HAServiceState.OBSERVER) {\n      throw new ServiceFailedException(\n          localTarget + \" is in observer state. \" +\n          \"Cannot be failover target\");\n    }\n  }\n\n  /**\n   * @return an {@link HAServiceTarget} for the current active node\n   * in the cluster, or null if no node is active.\n   * @throws IOException if a ZK-related issue occurs\n   * @throws InterruptedException if thread is interrupted \n   */\n  private HAServiceTarget getCurrentActive()\n      throws IOException, InterruptedException {\n    synchronized (elector) {\n      synchronized (this) {\n        byte[] activeData;\n        try {\n          activeData = elector.getActiveData();","sourceCodeStart":757,"sourceCodeEnd":793,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/ZKFailoverController.java#L757-L793","documentation":"Thrown by ZKFailoverController.checkEligibleForFailover() when a graceful failover is requested to this ZKFC's node while the local service state is HAServiceState.OBSERVER (an HDFS Observer NameNode). The health check runs first and passes, but observer nodes are deliberately excluded from becoming active, so the operation is refused with ServiceFailedException. This protects against promoting a read-only observer into the writer role.","triggerScenarios":"Running 'hdfs haadmin -failover <from> <to>' (or the ZKFCProtocol.gracefulFailover RPC on port 8019) where the ZKFC on <to> is managing a NameNode started with dfs.ha.nn.observer.<id>=true; also failover scripts that iterate over all NNs and pick the first healthy one, hitting the observer.","commonSituations":"Observer NameNode deployments (HDFS-12943) where an operator or automation tool targets the observer for failover; test clusters where nodes were converted to observers after the failover script was written.","solutions":["Pick a different target: run 'hdfs haadmin -getAllServiceState' and fail over to a node whose state is standby, not observer","If the observer node must become failover-capable, reconfigure and restart that NameNode as a standby (remove dfs.ha.nn.observer.<id>=true) so its ZKFC can elect it","Verify you are not hitting the sibling health failure: the message 'is not currently healthy' is checked before the observer check"],"exampleFix":"# before\nhdfs haadmin -failover nn1 nn2   # nn2 is an ObserverNode -> ServiceFailedException\n\n# after\nhdfs haadmin -getAllServiceState    # confirm states first\nhdfs haadmin -failover nn1 nn3      # target a standby node","handlingStrategy":"validation","validationCode":"HAServiceProtocol proxy = ...; // to the target NN\nHAServiceStatus status = proxy.getServiceStatus();\nif (status.getState() == HAServiceState.OBSERVER) {\n  throw new IllegalStateException(\"target is an observer; pick a standby node\");\n}\nif (status.getState() != HAServiceState.STANDBY) {\n  throw new IllegalStateException(\"target must be standby, was \" + status.getState());\n}","typeGuard":null,"tryCatchPattern":"try {\n  zfc.gracefulFailover(0); // ZKFCProtocol on target\n} catch (ServiceFailedException e) {\n  // message distinguishes 'not currently healthy' vs 'is in observer state'\n  LOG.warn(\"graceful failover refused: {}\", e.getMessage());\n}","preventionTips":["Failover automation should call hdfs haadmin -getAllServiceState first and filter for standby targets","Keep observer nodes out of failover target lists in runbooks and scripts","Alert when a node's HAServiceState changes so operators know the eligible target set"],"tags":["hadoop","hdfs","ha","failover","observer","zkfc"],"backgroundTag":"ha-failover-ineligible-target","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}