What causes "Parent path is not a directory: " + parent" in apache/hadoop?

Inspect the parent named in the message: ls -l shows a regular file where a directory is required.

apache/hadoop · error · ParentNotDirectoryException

Parent path is not a directory: " + parent

Error message

Parent path is not a directory: " + parent

What it means

mkdirsWithOptionalPermission throws ParentNotDirectoryException when the target's parent path exists on local disk but is a regular file, so directory creation cannot proceed. mkdirs can create missing directories but cannot replace a file with a directory. The message names the offending parent path.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/RawLocalFileSystem.java:911

  }

  @Override
  public boolean mkdirs(Path f, FsPermission permission) throws IOException {
    return mkdirsWithOptionalPermission(f, permission);
  }

  private boolean mkdirsWithOptionalPermission(Path f, FsPermission permission)
      throws IOException {
    if(f == null) {
      throw new IllegalArgumentException("mkdirs path arg is null");
    }
    Path parent = f.getParent();
    File p2f = pathToFile(f);
    File parent2f = null;
    if(parent != null) {
      parent2f = pathToFile(parent);
      if(parent2f != null && parent2f.exists() && !parent2f.isDirectory()) {
        throw new ParentNotDirectoryException("Parent path is not a directory: "
            + parent);
      }
    }
    if (p2f.exists() && !p2f.isDirectory()) {
      throw new FileAlreadyExistsException("Destination exists" +
              " and is not a directory: " + p2f.getCanonicalPath());
    }
    return (parent == null || parent2f.exists() || mkdirs(parent)) &&
      (mkOneDirWithMode(f, p2f, permission) || p2f.isDirectory());
  }
  
  
  @Override
  public Path getHomeDirectory() {
    return this.makeQualified(new Path(System.getProperty("user.home")));
  }

  /**

View on GitHub (pinned to 2add963021)

Solutions

Inspect the parent named in the message: ls -l shows a regular file where a directory is required.
Remove or relocate the blocking file: rm /data/out (or mv it elsewhere), then retry mkdirs.
Pick a non-colliding prefix in configuration so files and directory trees never share a path.
If concurrent create/mkdir may race, serialize with a lock or pre-create the directory layout once at startup.

Example fix

# before
$ ls -l /data/out
-rw-r--r-- 1 app app 1024 out    # file blocks parent dir
fs.mkdirs(new Path("/data/out/subdir")); // ParentNotDirectoryException

# after
$ rm /data/out
fs.mkdirs(new Path("/data/out/subdir")); // succeeds

Defensive patterns

Strategy: validation

Validate before calling

// ensure no ancestor of the target is a regular file
Path t = target;
while (t.getParent() != null) {
  t = t.getParent();
  if (fs.exists(t) && !fs.getFileStatus(t).isDirectory()) {
    throw new IOException("Ancestor " + t + " is a file; cannot mkdir under it");
  }
}
fs.mkdirs(target);

Try / catch

try {
  fs.mkdirs(dir);
} catch (ParentNotDirectoryException e) {
  throw new IOException("Path layout conflict under " + dir
      + ": remove the file occupying the parent path", e);
}

Prevention

Design path layouts so directory prefixes never collide with file paths.
Check ancestor types before mkdir when reusing paths across app versions.
Resolve symlinks before deciding layout; a link to a file blocks mkdir.

When it happens

Trigger: fs.mkdirs("/data/out/subdir") where /data/out is a regular file; interleaved operations where create() wrote a file at a path later used as a directory prefix; flattening/nesting path schemes between versions of an app.

Common situations: Path scheme changes (previously /data-out as a file, now /data/out/... as a tree) without cleanup, tests that create files then reuse prefixes as directories, symlink to a file occupying the parent slot, or config changed to nest paths under an existing file location.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/43fffa7b35d9f4d7. Report an issue: GitHub.