{"record":{"id":"e570cf7c43e0644b","repo":"apache/hadoop","slug":"the-namenode-is-configured-to-report-unhealthy-to","errorCode":null,"errorMessage":"The NameNode is configured to report UNHEALTHY to ZKFC in Safemode.","messagePattern":"The NameNode is configured to report UNHEALTHY to ZKFC in Safemode\\.","errorType":"exception","errorClass":"HealthCheckFailedException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java","lineNumber":2006,"sourceCode":"    String operationName = \"monitorHealth\";\n    namesystem.checkSuperuserPrivilege(operationName);\n    if (!haEnabled) {\n      return; // no-op, if HA is not enabled\n    }\n    long start = Time.monotonicNow();\n    getNamesystem().checkAvailableResources();\n    long end = Time.monotonicNow();\n    if (end - start >= HEALTH_MONITOR_WARN_THRESHOLD_MS) {\n      // log a warning if it take >= 5 seconds.\n      LOG.warn(\"Remote IP {} checking available resources took {}ms\",\n          Server.getRemoteIp(), end - start);\n    }\n    if (!getNamesystem().nameNodeHasResourcesAvailable()) {\n      throw new HealthCheckFailedException(\n          \"The NameNode has no resources available\");\n    }\n    if (notBecomeActiveInSafemode && isInSafeMode()) {\n      throw new HealthCheckFailedException(\"The NameNode is configured to \" +\n          \"report UNHEALTHY to ZKFC in Safemode.\");\n    }\n  }\n  \n  synchronized void transitionToActive() throws IOException {\n    String operationName = \"transitionToActive\";\n    namesystem.checkSuperuserPrivilege(operationName);\n    if (!haEnabled) {\n      throw new ServiceFailedException(\"HA for namenode is not enabled\");\n    }\n    if (state == OBSERVER_STATE) {\n      throw new ServiceFailedException(\n          \"Cannot transition from '\" + OBSERVER_STATE + \"' to '\" +\n              ACTIVE_STATE + \"'\");\n    }\n    if (notBecomeActiveInSafemode && isInSafeMode()) {\n      throw new ServiceFailedException(getRole() + \" still not leave safemode\");\n    }","sourceCodeStart":1988,"sourceCodeEnd":2024,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java#L1988-L2024","documentation":"monitorHealth throws HealthCheckFailedException when the NameNode is configured with dfs.ha.nn.not-become-active-in-safemode=true (notBecomeActiveInSafemode) and is currently in safemode. The feature deliberately reports UNHEALTHY to ZKFC while the NN has not left safemode, preventing a still-initializing NameNode from being promoted or kept active.","triggerScenarios":"ZKFC monitorHealth calls while notBecomeActiveInSafemode=true and isInSafeMode() is true - typically a safemode that has not exited because block reports are still arriving or blocks are missing.","commonSituations":"Cluster restart with large namespace where safemode exit is slow; safemode stuck on under-replicated/missing blocks after datanode loss; operator enabled the flag to stop ZKFC promoting a half-ready NN and now sees UNHEALTHY.","solutions":["Check safemode status ('hdfs dfsadmin -safemode get') and why it has not exited: inspect missing/under-replicated blocks ('hdfs fsck /'), wait for datanode block reports, or clear lost blocks.","Once safemode exits, the health check passes with no restart needed.","If promotion during safemode is actually acceptable in your environment, remove/disable dfs.ha.nn.not-become-active-in-safemode.","As a controlled last resort, 'hdfs dfsadmin -safemode leave' - only after verifying no data-loss risk."],"exampleFix":"<!-- before -->\n<property><name>dfs.ha.nn.not-become-active-in-safemode</name><value>true</value></property>\n# NN stuck in safemode -> ZKFC reports UNHEALTHY\n\n<!-- after -->\nhdfs dfsadmin -safemode get        # resolve missing blocks / wait for reports\nhdfs dfsadmin -safemode leave      # controlled exit once safe; health check then passes","handlingStrategy":"validation","validationCode":"// before health checks matter, know the safemode state\nif (namesystem.isInSafeMode() && notBecomeActiveInSafemodeEnabled(conf)) {\n  LOG.warn(\"NN will report UNHEALTHY to ZKFC until safemode exits\");\n}\n// CLI equivalent: hdfs dfsadmin -safemode get","typeGuard":"boolean isSafemodeHealthFailure(Throwable t) {\n  return t instanceof HealthCheckFailedException\n      && String.valueOf(t.getMessage()).contains(\"UNHEALTHY to ZKFC in Safemode\");\n}","tryCatchPattern":null,"preventionTips":["Alert on safemode duration, not just on the health failure it causes later.","Fix datanode availability/missing blocks promptly after restarts so safemode exits before ZKFC escalates.","Only set dfs.ha.nn.not-become-active-in-safemode when you accept UNHEALTHY-while-initializing semantics."],"tags":["hdfs","namenode","ha","safemode","health-check","zookeeper-failover"],"backgroundTag":"ha-health-check-failed","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}