apache/hadoop · critical · org.apache.hadoop.hdfs.server.common.InconsistentFSStateException

Directory {dir} is in an inconsistent state: storage directo

Error message

Directory {dir} is in an inconsistent state: storage directory does not exist or is not accessible.

What it means

In recoverStorageDirs(), StorageDirectory.analyzeStorage() returned NON_EXISTENT for a configured storage directory: the directory does not exist or is not accessible. Unlike the DataNode, the NameNode never creates directories on demand — any configured-but-missing directory is fatal (InconsistentFSStateException).

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java:393

    // sure that we properly examine their state, but we make sure we don't
    // mutate the shared dir below in the actual loop.
    for (Iterator<StorageDirectory> it = 
                      storage.dirIterator(); it.hasNext();) {
      StorageDirectory sd = it.next();
      StorageState curState;
      if (startOpt == StartupOption.METADATAVERSION) {
        /* All we need is the layout version. */
        storage.readProperties(sd);
        return true;
      }

      try {
        curState = sd.analyzeStorage(startOpt, storage);
        // sd is locked but not opened
        switch(curState) {
        case NON_EXISTENT:
          // name-node fails if any of the configured storage dirs are missing
          throw new InconsistentFSStateException(sd.getRoot(),
                      "storage directory does not exist or is not accessible.");
        case NOT_FORMATTED:
          break;
        case NORMAL:
          break;
        default:  // recovery is possible
          sd.doRecover(curState);
        }
        if (curState != StorageState.NOT_FORMATTED 
            && startOpt != StartupOption.ROLLBACK) {
          // read and verify consistency with other directories
          storage.readProperties(sd, startOpt);
          isFormatted = true;
        }
        if (startOpt == StartupOption.IMPORT && isFormatted)
          // import of a checkpoint is allowed only into empty image directories
          throw new IOException("Cannot import image from a checkpoint. " 
              + " NameNode already contains an image in " + sd.getRoot());

View on GitHub (pinned to 2add963021)

Solutions

1. Inspect the directory named in the message: ls -ld <dir>, check the mount (df -h) and ownership.
2. Create it and fix ownership/perms: mkdir -p <dir> && chown hdfs:hdfs <dir> && chmod 700 <dir>.
3. Correct the config if the path is a typo (hdfs getconf -confKey dfs.namenode.name.dir).
4. If the volume is permanently gone, restore its metadata from a sibling dir or backup — or remove the entry from config only if other dirs hold copies of the metadata.

Example fix

# before
ls -ld /dfs/nn1   # No such file or directory

# after
mkdir -p /dfs/nn1
chown hdfs:hdfs /dfs/nn1
chmod 700 /dfs/nn1
hdfs --daemon start namenode

for (URI u : allConfiguredDirs) {
  Path p = Paths.get(u);
  if (!Files.isDirectory(p)) {
    fail(u + " does not exist");
  } else if (!Files.isReadable(p) || !Files.isWritable(p)) {
    fail(u + " not accessible");
  }
}

When it happens

Trigger: A dfs.namenode.name.dir / edits.dir entry deleted, never created, on an unmounted volume, or unreadable due to permissions; path typo in hdfs-site.xml; NFS or aggregate volume unavailable at boot.

Common situations: Disk replaced or not mounted after reboot; directory removed by cleanup jobs; container/pod started without the hostPath volume; ownership/ACL changes after a user rename.

Related errors

• Cannot lock storage {root}. The directory is already locked
• All specified directories are not accessible or do not exist
• NameNode is not formatted.
• Unknown nameservice: {}
• Configuration has multiple addresses that match local node's

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/5d167741026ee14d. Report an issue: GitHub.