{"record":{"id":"03648a4d41ab1ff0","repo":"apache/hadoop","slug":"the-namenode-has-no-resources-available","errorCode":null,"errorMessage":"The NameNode has no resources available","messagePattern":"The NameNode has no resources available","errorType":"exception","errorClass":"HealthCheckFailedException","httpStatus":null,"severity":"critical","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java","lineNumber":2002,"sourceCode":"    }\n  }\n\n  synchronized void monitorHealth() throws IOException {\n    String operationName = \"monitorHealth\";\n    namesystem.checkSuperuserPrivilege(operationName);\n    if (!haEnabled) {\n      return; // no-op, if HA is not enabled\n    }\n    long start = Time.monotonicNow();\n    getNamesystem().checkAvailableResources();\n    long end = Time.monotonicNow();\n    if (end - start >= HEALTH_MONITOR_WARN_THRESHOLD_MS) {\n      // log a warning if it take >= 5 seconds.\n      LOG.warn(\"Remote IP {} checking available resources took {}ms\",\n          Server.getRemoteIp(), end - start);\n    }\n    if (!getNamesystem().nameNodeHasResourcesAvailable()) {\n      throw new HealthCheckFailedException(\n          \"The NameNode has no resources available\");\n    }\n    if (notBecomeActiveInSafemode && isInSafeMode()) {\n      throw new HealthCheckFailedException(\"The NameNode is configured to \" +\n          \"report UNHEALTHY to ZKFC in Safemode.\");\n    }\n  }\n  \n  synchronized void transitionToActive() throws IOException {\n    String operationName = \"transitionToActive\";\n    namesystem.checkSuperuserPrivilege(operationName);\n    if (!haEnabled) {\n      throw new ServiceFailedException(\"HA for namenode is not enabled\");\n    }\n    if (state == OBSERVER_STATE) {\n      throw new ServiceFailedException(\n          \"Cannot transition from '\" + OBSERVER_STATE + \"' to '\" +\n              ACTIVE_STATE + \"'\");","sourceCodeStart":1984,"sourceCodeEnd":2020,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java#L1984-L2020","documentation":"NameNode.monitorHealth (the HAServiceProtocol health check driven by ZKFC) calls FSNamesystem.checkAvailableResources() and then nameNodeHasResourcesAvailable(); when required storage resources are no longer available, it throws HealthCheckFailedException. A failed health check makes ZKFC mark the NN unhealthy and, if it was active, fence it and fail over.","triggerScenarios":"ZKFC's monitorHealth RPC at its check interval while the NameNode's required name/edit storage directories have failed (IO errors, mount loss, permission loss), so nameNodeHasResourcesAvailable() returns false; the code also warns when the resource check itself takes >= 5s.","commonSituations":"Name-directory disk failure or full volume; all journal edits directories (including QJM) unreachable; storage mounts dropped after host maintenance; NN left running with dead storage while ZKFC polls it.","solutions":["Check the NameNode log for storage-directory failed/resources messages ('failed storage directory', NameDirStatuses in JMX) and restore the underlying storage: fix mounts, permissions, free space.","Restart the NameNode after repairing storage so failed directories re-register; in HA the standby/other NN should have taken over - confirm service continuity via 'hdfs haadmin -getAllServiceState'.","If directories are permanently lost, recover the namespace from the peer NN, a checkpoint/backup, or reformat (data-loss decision) before rejoining the cluster.","Investigate why resources became unavailable (disk health, ENOSPC alerts) to prevent immediate recurrence."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"# before trusting a NN, inspect its storage-dir health via JMX\nstatus=$(curl -s http://nn-host:9870/jmx | jq -r '.beans[] | select(.name==\"Hadoop:service=NameNode,name=NameNodeInfo\") | .NameDirStatuses')\necho \"$status\" | grep -q '\"failed\"' && echo \"storage dirs failed - health check will fail\"","typeGuard":"boolean isHealthCheckFailure(Throwable t) {\n  return t instanceof org.apache.hadoop.ha.HealthCheckFailedException;\n}","tryCatchPattern":"try {\n  nn.monitorHealth();\n} catch (HealthCheckFailedException e) {\n  // NN has no resources: trigger/verify failover instead of retrying blindly\n  failoverToHealthyPeer(); // then page: storage must be repaired before the NN can return\n}","preventionTips":["Monitor NameDirStatuses JMX and disk space on name/edit volumes so you catch failing dirs before ZKFC does.","Use multiple name dirs on independent volumes and QJM journalnodes so a single disk cannot exhaust required resources.","Automate fencing verification: after any health-check failure, confirm the old active is fenced before reusing hosts."],"tags":["hdfs","namenode","ha","zookeeper-failover","health-check","storage-failure"],"backgroundTag":"ha-health-check-failed","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}