apache/hadoop · critical · HealthCheckFailedException

The NameNode has no resources available

Error message

The NameNode has no resources available

What it means

NameNode.monitorHealth (the HAServiceProtocol health check driven by ZKFC) calls FSNamesystem.checkAvailableResources() and then nameNodeHasResourcesAvailable(); when required storage resources are no longer available, it throws HealthCheckFailedException. A failed health check makes ZKFC mark the NN unhealthy and, if it was active, fence it and fail over.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java:2002

    }
  }

  synchronized void monitorHealth() throws IOException {
    String operationName = "monitorHealth";
    namesystem.checkSuperuserPrivilege(operationName);
    if (!haEnabled) {
      return; // no-op, if HA is not enabled
    }
    long start = Time.monotonicNow();
    getNamesystem().checkAvailableResources();
    long end = Time.monotonicNow();
    if (end - start >= HEALTH_MONITOR_WARN_THRESHOLD_MS) {
      // log a warning if it take >= 5 seconds.
      LOG.warn("Remote IP {} checking available resources took {}ms",
          Server.getRemoteIp(), end - start);
    }
    if (!getNamesystem().nameNodeHasResourcesAvailable()) {
      throw new HealthCheckFailedException(
          "The NameNode has no resources available");
    }
    if (notBecomeActiveInSafemode && isInSafeMode()) {
      throw new HealthCheckFailedException("The NameNode is configured to " +
          "report UNHEALTHY to ZKFC in Safemode.");
    }
  }
  
  synchronized void transitionToActive() throws IOException {
    String operationName = "transitionToActive";
    namesystem.checkSuperuserPrivilege(operationName);
    if (!haEnabled) {
      throw new ServiceFailedException("HA for namenode is not enabled");
    }
    if (state == OBSERVER_STATE) {
      throw new ServiceFailedException(
          "Cannot transition from '" + OBSERVER_STATE + "' to '" +
              ACTIVE_STATE + "'");

View on GitHub (pinned to 2add963021)

Solutions

1. Check the NameNode log for storage-directory failed/resources messages ('failed storage directory', NameDirStatuses in JMX) and restore the underlying storage: fix mounts, permissions, free space.
2. Restart the NameNode after repairing storage so failed directories re-register; in HA the standby/other NN should have taken over - confirm service continuity via 'hdfs haadmin -getAllServiceState'.
3. If directories are permanently lost, recover the namespace from the peer NN, a checkpoint/backup, or reformat (data-loss decision) before rejoining the cluster.
4. Investigate why resources became unavailable (disk health, ENOSPC alerts) to prevent immediate recurrence.

# before trusting a NN, inspect its storage-dir health via JMX
status=$(curl -s http://nn-host:9870/jmx | jq -r '.beans[] | select(.name=="Hadoop:service=NameNode,name=NameNodeInfo") | .NameDirStatuses')
echo "$status" | grep -q '"failed"' && echo "storage dirs failed - health check will fail"

boolean isHealthCheckFailure(Throwable t) {
  return t instanceof org.apache.hadoop.ha.HealthCheckFailedException;
}

try {
  nn.monitorHealth();
} catch (HealthCheckFailedException e) {
  // NN has no resources: trigger/verify failover instead of retrying blindly
  failoverToHealthyPeer(); // then page: storage must be repaired before the NN can return
}

When it happens

Trigger: ZKFC's monitorHealth RPC at its check interval while the NameNode's required name/edit storage directories have failed (IO errors, mount loss, permission loss), so nameNodeHasResourcesAvailable() returns false; the code also warns when the resource check itself takes >= 5s.

Common situations: Name-directory disk failure or full volume; all journal edits directories (including QJM) unreachable; storage mounts dropped after host maintenance; NN left running with dead storage while ZKFC polls it.

Related errors

• The NameNode is configured to report UNHEALTHY to ZKFC in Sa
• Configuration has multiple addresses that match local node's
• Configuration dfs.namenode.rpc-address must be suffixed with
• Unexpected HAServiceStateProto:
• " + idHelpText + " mismatch: previously connected to " + idH

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/03648a4d41ab1ff0. Report an issue: GitHub.