What causes "Not a directory: {}" in apache/hadoop?

Delete or rename the file occupying the parent path: fs.delete(parent, false).

apache/hadoop · error · FileAlreadyExistsException

Not a directory: {}

Error message

Not a directory: {}

What it means

createNonRecursive checks the immediate parent of the file being created; if getFileStatus(parent) resolves to an existing FILE rather than a directory, it throws FileAlreadyExistsException('Not a directory: <parent>'). If the parent is missing entirely you get FileNotFoundException from getFileStatus instead, so this specific message means the parent exists but has the wrong type.

Source

Thrown at hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/CosNFileSystem.java:285

      boolean recursive) throws PathIOException {
    if (isEmptyDir) {
      return true;
    }
    if (recursive) {
      return false;
    } else {
      throw new PathIOException(this.bucket, "Can not delete root path");
    }
  }

  @Override
  public FSDataOutputStream createNonRecursive(Path f, FsPermission permission,
      EnumSet<CreateFlag> flags, int bufferSize, short replication,
      long blockSize, Progressable progress) throws IOException {
    Path parent = f.getParent();
    if (null != parent) {
      if (!getFileStatus(parent).isDirectory()) {
        throw new FileAlreadyExistsException("Not a directory: " + parent);
      }
    }

    return create(f, permission, flags.contains(CreateFlag.OVERWRITE),
        bufferSize, replication, blockSize, progress);
  }

  @Override
  public boolean delete(Path f, boolean recursive) throws IOException {
    LOG.debug("Ready to delete path: [{}]. recursive: [{}].", f, recursive);
    FileStatus status;
    try {
      status = getFileStatus(f);
    } catch (FileNotFoundException e) {
      LOG.debug("Ready to delete the file: [{}], but it does not exist.", f);
      return false;
    }
    Path absolutePath = makeAbsolute(f);

View on GitHub (pinned to 2add963021)

Solutions

Delete or rename the file occupying the parent path: fs.delete(parent, false).
If the parent may be missing, call fs.mkdirs(parent) first.
Switch to fs.create(), which creates parent directories implicitly.

Example fix

// before
out = fs.createNonRecursive(new Path('/logs/2026/part-0'), perms, flags, buf, rep, block, null);
// throws FileAlreadyExistsException: Not a directory: /logs/2026

// after
Path parent = new Path('/logs/2026');
if (fs.exists(parent) && fs.getFileStatus(parent).isFile()) {
  fs.delete(parent, false);
}
fs.mkdirs(parent);
out = fs.createNonRecursive(new Path('/logs/2026/part-0'), perms, flags, buf, rep, block, null);

Defensive patterns

Strategy: validation

Validate before calling

Path parent = f.getParent();
if (parent != null) {
  try {
    if (!fs.getFileStatus(parent).isDirectory()) {
      throw new IllegalStateException('Parent exists but is a file: ' + parent);
    }
  } catch (FileNotFoundException e) {
    fs.mkdirs(parent);
  }
}

Try / catch

try {
  fs.createNonRecursive(f, perms, flags, buf, rep, block, null);
} catch (FileAlreadyExistsException e) {
  // 'Not a directory: <parent>' -> clear the conflicting file and retry once
  fs.delete(f.getParent(), false);
  fs.mkdirs(f.getParent());
}

Prevention

Create the parent directory chain before non-recursive creates
Never let the same key be both a file and a directory level in object stores
Treat 'Not a directory' as layout damage: audit how the file got there

When it happens

Trigger: fs.createNonRecursive(new Path('/a/b/file'), ...) where /a/b exists as a file object; writers that skip mkdirs and assume parents are directories.

Common situations: An earlier job wrote a file where a directory level is now needed (easy in object stores with no real directories); a partition path collides with an existing file; concurrent producer created the parent as a file between checks.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/c1faad4b2ebd0707. Report an issue: GitHub.