apache/hadoop · error · HealthCheckFailedException

The NameNode is configured to report UNHEALTHY to ZKFC in Sa

Error message

The NameNode is configured to report UNHEALTHY to ZKFC in Safemode.

What it means

monitorHealth throws HealthCheckFailedException when the NameNode is configured with dfs.ha.nn.not-become-active-in-safemode=true (notBecomeActiveInSafemode) and is currently in safemode. The feature deliberately reports UNHEALTHY to ZKFC while the NN has not left safemode, preventing a still-initializing NameNode from being promoted or kept active.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java:2006

    String operationName = "monitorHealth";
    namesystem.checkSuperuserPrivilege(operationName);
    if (!haEnabled) {
      return; // no-op, if HA is not enabled
    }
    long start = Time.monotonicNow();
    getNamesystem().checkAvailableResources();
    long end = Time.monotonicNow();
    if (end - start >= HEALTH_MONITOR_WARN_THRESHOLD_MS) {
      // log a warning if it take >= 5 seconds.
      LOG.warn("Remote IP {} checking available resources took {}ms",
          Server.getRemoteIp(), end - start);
    }
    if (!getNamesystem().nameNodeHasResourcesAvailable()) {
      throw new HealthCheckFailedException(
          "The NameNode has no resources available");
    }
    if (notBecomeActiveInSafemode && isInSafeMode()) {
      throw new HealthCheckFailedException("The NameNode is configured to " +
          "report UNHEALTHY to ZKFC in Safemode.");
    }
  }
  
  synchronized void transitionToActive() throws IOException {
    String operationName = "transitionToActive";
    namesystem.checkSuperuserPrivilege(operationName);
    if (!haEnabled) {
      throw new ServiceFailedException("HA for namenode is not enabled");
    }
    if (state == OBSERVER_STATE) {
      throw new ServiceFailedException(
          "Cannot transition from '" + OBSERVER_STATE + "' to '" +
              ACTIVE_STATE + "'");
    }
    if (notBecomeActiveInSafemode && isInSafeMode()) {
      throw new ServiceFailedException(getRole() + " still not leave safemode");
    }

View on GitHub (pinned to 2add963021)

Solutions

1. Check safemode status ('hdfs dfsadmin -safemode get') and why it has not exited: inspect missing/under-replicated blocks ('hdfs fsck /'), wait for datanode block reports, or clear lost blocks.
2. Once safemode exits, the health check passes with no restart needed.
3. If promotion during safemode is actually acceptable in your environment, remove/disable dfs.ha.nn.not-become-active-in-safemode.
4. As a controlled last resort, 'hdfs dfsadmin -safemode leave' - only after verifying no data-loss risk.

Example fix

<!-- before -->
<property><name>dfs.ha.nn.not-become-active-in-safemode</name><value>true</value></property>
# NN stuck in safemode -> ZKFC reports UNHEALTHY

<!-- after -->
hdfs dfsadmin -safemode get        # resolve missing blocks / wait for reports
hdfs dfsadmin -safemode leave      # controlled exit once safe; health check then passes

// before health checks matter, know the safemode state
if (namesystem.isInSafeMode() && notBecomeActiveInSafemodeEnabled(conf)) {
  LOG.warn("NN will report UNHEALTHY to ZKFC until safemode exits");
}
// CLI equivalent: hdfs dfsadmin -safemode get

boolean isSafemodeHealthFailure(Throwable t) {
  return t instanceof HealthCheckFailedException
      && String.valueOf(t.getMessage()).contains("UNHEALTHY to ZKFC in Safemode");
}

When it happens

Trigger: ZKFC monitorHealth calls while notBecomeActiveInSafemode=true and isInSafeMode() is true - typically a safemode that has not exited because block reports are still arriving or blocks are missing.

Common situations: Cluster restart with large namespace where safemode exit is slow; safemode stuck on under-replicated/missing blocks after datanode loss; operator enabled the flag to stop ZKFC promoting a half-ready NN and now sees UNHEALTHY.

Related errors

• The NameNode has no resources available
• {getRole()} still not leave safemode
• Configuration has multiple addresses that match local node's
• Configuration dfs.namenode.rpc-address must be suffixed with
• Unexpected HAServiceStateProto:

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/e570cf7c43e0644b. Report an issue: GitHub.