What causes "Directory does not exist: {}" in apache/hadoop?

Create the directory first (mkdirs) before operations that require it as a parent

apache/hadoop · error · FileNotFoundException

Directory does not exist: {}

Error message

Directory does not exist: {}

What it means

INodeDirectory.valueOf(INode, path) throws FileNotFoundException("Directory does not exist: <path>") when the resolved INode is null, meaning the path is absent from the namespace. This is the NameNode-internal cast helper used wherever a path component must be a directory (rename parents, quota updates, listings), so the exception surfaces through the RPC layer as a client-visible 'no such file or directory'.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java:61

import org.apache.hadoop.hdfs.util.ReadOnlyList;

import org.apache.hadoop.classification.VisibleForTesting;
import org.apache.hadoop.util.Preconditions;
import org.apache.hadoop.security.AccessControlException;

import static org.apache.hadoop.hdfs.protocol.HdfsConstants.BLOCK_STORAGE_POLICY_ID_UNSPECIFIED;

/**
 * Directory INode class.
 */
public class INodeDirectory extends INodeWithAdditionalFields
    implements INodeDirectoryAttributes {

  /** Cast INode to INodeDirectory. */
  public static INodeDirectory valueOf(INode inode, Object path
      ) throws FileNotFoundException, PathIsNotDirectoryException {
    if (inode == null) {
      throw new FileNotFoundException("Directory does not exist: "
          + DFSUtil.path2String(path));
    }
    if (!inode.isDirectory()) {
      throw new PathIsNotDirectoryException(DFSUtil.path2String(path));
    }
    return inode.asDirectory(); 
  }

  // Profiling shows that most of the file lists are between 1 and 4 elements.
  // Thus allocate the corresponding ArrayLists with a small initial capacity.
  public static final int DEFAULT_FILES_PER_DIRECTORY = 2;

  static final byte[] ROOT_NAME = DFSUtil.string2Bytes("");

  private List<INode> children = null;
  
  /** constructor */
  public INodeDirectory(long id, byte[] name, PermissionStatus permissions,

View on GitHub (pinned to 2add963021)

Solutions

Create the directory first (mkdirs) before operations that require it as a parent
When writing NN-side code, re-resolve via FSDirectory.getINode and handle null explicitly instead of calling valueOf blindly
For concurrency races, catch FileNotFoundException and either retry the whole operation or report the deletion to the caller

Example fix

// before
INodeDirectory dir = INodeDirectory.valueOf(fsDir.getINode(path, DirOp.READ), path);

// after
INode node = fsDir.getINode(path, DirOp.READ);
if (node == null) {
  throw new FileNotFoundException("Directory does not exist: " + path);
}
INodeDirectory dir = INodeDirectory.valueOf(node, path);

Defensive patterns

Strategy: validation

Validate before calling

// NN-side: resolve first, distinguish missing from wrong-type
INode node = fsDir.getINode(path, DirOp.READ);
if (node == null) {
  // missing: create it or report precisely
  throw new FileNotFoundException("Directory does not exist: " + DFSUtil.path2String(path));
}
INodeDirectory dir = INodeDirectory.valueOf(node, path);

Type guard

static boolean isExistingDirectory(FSDirectory fsDir, String path) {
  final INode n;
  try {
    fsDir.readLock();
    n = fsDir.getINode(path, DirOp.READ);
  } finally {
    fsDir.readUnlock();
  }
  return n != null && n.isDirectory();
}

Try / catch

catch (FileNotFoundException e) {
  // path vanished (possibly deleted concurrently): decide recreate vs report
  if (isRetryableRace(e)) { retryOperation(); } else { reportMissingPath(e); }
}

Prevention

mkdirs() parents before rename/create flows that assume they exist
Never cache directory paths across long-lived operations; re-resolve per request
Make clients idempotent so delete races surface as clean retries, not failures

When it happens

Trigger: An internal FSDirectory operation resolves a parent/target directory that was concurrently deleted; calling valueOf on a path that was never created; snapshot/rollback leaving code with a stale path string.

Common situations: Check-then-act races where a client verifies existence and another client deletes the directory before the operation lands; application code assuming a parent directory exists without mkdirs; replaying captured operations against a changed namespace.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/e2ee2b8827d1e806. Report an issue: GitHub.