apache/hadoop · error · InconsistentFSStateException

checkpoint directory does not exist or is not accessible.

Error message

checkpoint directory does not exist or is not accessible.

What it means

SecondaryNameNode throws InconsistentFSStateException when StorageDirectory.analyzeStorage(StartupOption.REGULAR) returns NON_EXISTENT: after the (non-throwing) mkdirs() attempt the checkpoint directory still does not exist, so it is either uncreatable or inaccessible. This is the common 'checkpoint dir missing' failure that stops the 2NN at startup.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SecondaryNameNode.java:1004

        }
        if(!isAccessible)
          throw new InconsistentFSStateException(sd.getRoot(),
              "cannot access checkpoint directory.");
        
        if (format) {
          // Don't confirm, since this is just the secondary namenode.
          LOG.info("Formatting storage directory " + sd);
          sd.clearDirectory();
        }
        
        StorageState curState;
        try {
          curState = sd.analyzeStorage(HdfsServerConstants.StartupOption.REGULAR, storage);
          // sd is locked but not opened
          switch(curState) {
          case NON_EXISTENT:
            // fail if any of the configured checkpoint dirs are inaccessible 
            throw new InconsistentFSStateException(sd.getRoot(),
                  "checkpoint directory does not exist or is not accessible.");
          case NOT_FORMATTED:
            break;  // it's ok since initially there is no current and VERSION
          case NORMAL:
            // Read the VERSION file. This verifies that:
            // (a) the VERSION file for each of the directories is the same,
            // and (b) when we connect to a NN, we can verify that the remote
            // node matches the same namespace that we ran on previously.
            storage.readProperties(sd);
            break;
          default:  // recovery is possible
            sd.doRecover(curState);
          }
        } catch (IOException ioe) {
          sd.unlock();
          throw ioe;
        }
      }

View on GitHub (pinned to 2add963021)

Solutions

Create the directory manually with the right ownership: mkdir -p <dir> && chown <2nn-user> <dir>, then restart the 2NN
Check the parent chain of the configured path is writable by the 2NN user and that no regular file shadows any path component
Verify the mount options (rw, not ro) of the filesystem holding the checkpoint dir
Correct a mistyped dfs.namenode.checkpoint.dir value in hdfs-site.xml

Example fix

# before
$ hdfs secondarynamenode  # InconsistentFSStateException: checkpoint directory does not exist or is not accessible.
# after
$ sudo mkdir -p /data/2nn/checkpoint
$ sudo chown hdfs:hadoop /data/2nn/checkpoint
$ hdfs secondarynamenode

Defensive patterns

Strategy: validation

Validate before calling

for (String dir : conf.getTrimmedStrings(DFSConfigKeys.DFS_NAMENODE_CHECKPOINT_DIR_KEY)) {
  File f = new File(dir);
  File parent = f.getParentFile();
  boolean creatable = (f.exists() && f.isDirectory())
      || (parent != null && parent.canWrite() && parent.isDirectory());
  if (!creatable) {
    throw new IOException("Checkpoint dir missing and not creatable: " + dir);
  }
}

Try / catch

catch (InconsistentFSStateException e) {
  if (e.getMessage().contains("does not exist or is not accessible")) {
    // create/repair the directory out-of-band, then restart the 2NN once
  }
}

Prevention

Create checkpoint directories during node provisioning and verify mounts are rw before starting the 2NN
Add a pre-flight check script (df/mount/ls) to 2NN service start (systemd ExecStartPre)
Monitor for silently failing mkdirs by checking directory existence after any storage-related change

When it happens

Trigger: analyzeStorage() finds neither a usable directory state (NORMAL/NOT_FORMATTED/recoverable) nor an existing root — typically because mkdirs() silently returned false (read-only mount, missing parent, parent path occupied by a regular file) or permissions prevent creation.

Common situations: dfs.namenode.checkpoint.dir points to a path on an NFS mount mounted read-only; the directory was deleted or never created on a new host; a file exists where the directory should be; the 2NN user lacks create permission on the parent.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/29f366ffdde992cc. Report an issue: GitHub.