apache/hadoop · error · InconsistentFSStateException

cannot access checkpoint directory.

Error message

cannot access checkpoint directory.

What it means

SecondaryNameNode throws InconsistentFSStateException during checkpoint-storage setup when sd.getRoot().mkdirs() raises a SecurityException for a configured checkpoint directory. It means the 2NN process is not permitted to create or touch the directory configured under dfs.namenode.checkpoint.dir, so startup aborts before any checkpoint can run.

Solutions

1. Grant the 2NN process write access to every dfs.namenode.checkpoint.dir path (chown/chmod the directories to the user running the SecondaryNameNode)
2. Verify the configured path is correct in hdfs-site.xml and points to a location the process may create
3. If a SecurityManager policy file is in use, add grant FilePermission "<checkpoint.dir>/*", "write,delete" (and read) for the 2NN principal
4. Restart the SecondaryNameNode and confirm it reaches the 'Secondary NameNode is up' log line

Example fix

# before: checkpoint dir owned by another user, policy denies write
ls -ld /data/2nn/checkpoint   # drwxr-x--- hdfs hdfs
# after
chown -R hdfs2nn:hadoop /data/2nn/checkpoint
chmod 755 /data/2nn/checkpoint

// before starting the SecondaryNameNode
String dir = conf.get(DFSConfigKeys.DFS_NAMENODE_CHECKPOINT_DIR_KEY,
    DFSConfigKeys.DFS_NAMENODE_CHECKPOINT_DIR_DEFAULT);
File f = new File(dir);
if (!f.exists() && !f.mkdirs()) {
  throw new IOException("Checkpoint dir not creatable: " + dir
      + " — fix permissions/ownership first");
}
try (FileOutputStream probe = new FileOutputStream(new File(f, ".rw_probe"), true)) {
  // write access confirmed
} catch (SecurityException | IOException e) {
  throw new IOException("No write access to checkpoint dir " + dir, e);
}

try {
  secondary = new SecondaryNameNode(conf);
} catch (InconsistentFSStateException e) {
  // e.g. cannot access checkpoint directory — report sd root from the message
  LOG.error("2NN storage inaccessible: {}", e.getMessage());
  // do not retry in a loop; fix filesystem permissions first
}

When it happens

Trigger: Constructing/starting SecondaryNameNode: its storage loop calls sd.getRoot().mkdirs() on each checkpoint dir; if a SecurityManager policy (or sandboxed JVM) denies filesystem access to that path, isAccessible becomes false and this exception is thrown. Note that mkdirs() merely returning false is ignored here; only the throwing SecurityException path triggers it.

Common situations: Running the 2NN under a different user than the one owning the checkpoint directories while a Java security policy is in effect; a restrictive policy file granting no FilePermission write on the checkpoint path; checkpoint dir on a protected/ACL-restricted mount after a user or policy change.

Understand the failure class

Background: Permission denied / not authorized / 403 Forbidden: access-control rejections when the caller lacks the required role, grant, or ownership — this error's family across 18 libraries.

Related errors

• checkpoint directory does not exist or is not accessible.
• Cannot create dump directory
• Cannot import image from a checkpoint. NameNode already…
• Cannot remove current dump directory
• Could not create directory

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/7fe71827e1f0eebd. Report an issue: GitHub.

Appendix: source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SecondaryNameNode.java:988

     * @throws IOException
     */
    void recoverCreate(boolean format) throws IOException {
      storage.attemptRestoreRemovedStorage();
      storage.unlockAll();

      for (Iterator<StorageDirectory> it = 
                   storage.dirIterator(); it.hasNext();) {
        StorageDirectory sd = it.next();
        boolean isAccessible = true;
        try { // create directories if don't exist yet
          if(!sd.getRoot().mkdirs()) {
            // do nothing, directory is already created
          }
        } catch(SecurityException se) {
          isAccessible = false;
        }
        if(!isAccessible)
          throw new InconsistentFSStateException(sd.getRoot(),
              "cannot access checkpoint directory.");
        
        if (format) {
          // Don't confirm, since this is just the secondary namenode.
          LOG.info("Formatting storage directory " + sd);
          sd.clearDirectory();
        }
        
        StorageState curState;
        try {
          curState = sd.analyzeStorage(HdfsServerConstants.StartupOption.REGULAR, storage);
          // sd is locked but not opened
          switch(curState) {
          case NON_EXISTENT:
            // fail if any of the configured checkpoint dirs are inaccessible 
            throw new InconsistentFSStateException(sd.getRoot(),
                  "checkpoint directory does not exist or is not accessible.");
          case NOT_FORMATTED:

View on GitHub (pinned to 2add963021)