What causes "cannot access checkpoint directory." in apache/hadoop?

Grant the 2NN process write access to every dfs.namenode.checkpoint.dir path (chown/chmod the directories to the user running the SecondaryNameNode)

apache/hadoop · error · InconsistentFSStateException

cannot access checkpoint directory.

Error message

cannot access checkpoint directory.

What it means

SecondaryNameNode throws InconsistentFSStateException during checkpoint-storage setup when sd.getRoot().mkdirs() raises a SecurityException for a configured checkpoint directory. It means the 2NN process is not permitted to create or touch the directory configured under dfs.namenode.checkpoint.dir, so startup aborts before any checkpoint can run.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SecondaryNameNode.java:988

     * @throws IOException
     */
    void recoverCreate(boolean format) throws IOException {
      storage.attemptRestoreRemovedStorage();
      storage.unlockAll();

      for (Iterator<StorageDirectory> it = 
                   storage.dirIterator(); it.hasNext();) {
        StorageDirectory sd = it.next();
        boolean isAccessible = true;
        try { // create directories if don't exist yet
          if(!sd.getRoot().mkdirs()) {
            // do nothing, directory is already created
          }
        } catch(SecurityException se) {
          isAccessible = false;
        }
        if(!isAccessible)
          throw new InconsistentFSStateException(sd.getRoot(),
              "cannot access checkpoint directory.");
        
        if (format) {
          // Don't confirm, since this is just the secondary namenode.
          LOG.info("Formatting storage directory " + sd);
          sd.clearDirectory();
        }
        
        StorageState curState;
        try {
          curState = sd.analyzeStorage(HdfsServerConstants.StartupOption.REGULAR, storage);
          // sd is locked but not opened
          switch(curState) {
          case NON_EXISTENT:
            // fail if any of the configured checkpoint dirs are inaccessible 
            throw new InconsistentFSStateException(sd.getRoot(),
                  "checkpoint directory does not exist or is not accessible.");
          case NOT_FORMATTED:

View on GitHub (pinned to 2add963021)

Solutions

Grant the 2NN process write access to every dfs.namenode.checkpoint.dir path (chown/chmod the directories to the user running the SecondaryNameNode)
Verify the configured path is correct in hdfs-site.xml and points to a location the process may create
If a SecurityManager policy file is in use, add grant FilePermission "<checkpoint.dir>/*", "write,delete" (and read) for the 2NN principal
Restart the SecondaryNameNode and confirm it reaches the 'Secondary NameNode is up' log line

Example fix

# before: checkpoint dir owned by another user, policy denies write
ls -ld /data/2nn/checkpoint   # drwxr-x--- hdfs hdfs
# after
chown -R hdfs2nn:hadoop /data/2nn/checkpoint
chmod 755 /data/2nn/checkpoint

Defensive patterns

Strategy: validation

Validate before calling

// before starting the SecondaryNameNode
String dir = conf.get(DFSConfigKeys.DFS_NAMENODE_CHECKPOINT_DIR_KEY,
    DFSConfigKeys.DFS_NAMENODE_CHECKPOINT_DIR_DEFAULT);
File f = new File(dir);
if (!f.exists() && !f.mkdirs()) {
  throw new IOException("Checkpoint dir not creatable: " + dir
      + " — fix permissions/ownership first");
}
try (FileOutputStream probe = new FileOutputStream(new File(f, ".rw_probe"), true)) {
  // write access confirmed
} catch (SecurityException | IOException e) {
  throw new IOException("No write access to checkpoint dir " + dir, e);
}

Try / catch

try {
  secondary = new SecondaryNameNode(conf);
} catch (InconsistentFSStateException e) {
  // e.g. cannot access checkpoint directory — report sd root from the message
  LOG.error("2NN storage inaccessible: {}", e.getMessage());
  // do not retry in a loop; fix filesystem permissions first
}

Prevention

Provision checkpoint directories with the 2NN user as owner in your host bootstrap (Ansible/Puppet), not at runtime
Alert on 2NN startup failure so permission drift on checkpoint dirs is caught immediately
Keep Java security policies, when used, in the same config management as the checkpoint paths

When it happens

Trigger: Constructing/starting SecondaryNameNode: its storage loop calls sd.getRoot().mkdirs() on each checkpoint dir; if a SecurityManager policy (or sandboxed JVM) denies filesystem access to that path, isAccessible becomes false and this exception is thrown. Note that mkdirs() merely returning false is ignored here; only the throwing SecurityException path triggers it.

Common situations: Running the 2NN under a different user than the one owning the checkpoint directories while a Java security policy is in effect; a restrictive policy file granting no FilePermission write on the checkpoint path; checkpoint dir on a protected/ACL-restricted mount after a user or policy change.

Understand the failure class

Background: Permission denied / not authorized / 403 Forbidden: access-control rejections when the caller lacks the required role, grant, or ownership — this error's family across 18 libraries.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/7fe71827e1f0eebd. Report an issue: GitHub.