apache/hadoop · error · IOException

mkdir of ${f} failed

Error message

mkdir of ${f} failed

What it means

The tail of primitiveMkdir: after preconditions pass it calls this.mkdirs(f, permission) and treats a false return as IOException('mkdir of f failed'). mkdirs returns false with no detail, so the real cause - usually an existing FILE at f or a permission problem on the parent - must be found by checking those directly.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java:1413

    throws IOException {

    if (!createParent) { // parent must exist.
      // since the this.mkdirs makes parent dirs automatically
      // we must throw exception if parent does not exist.
      final FileStatus stat = getFileStatus(f.getParent());
      if (stat == null) {
        throw new FileNotFoundException("Missing parent:" + f);
      }
      if (!stat.isDirectory()) {
        throw new ParentNotDirectoryException("parent is not a dir");
      }
      // parent does exist - go ahead with mkdir of leaf
    }
    // Default impl is to assume that permissions do not matter and hence
    // calling the regular mkdirs is good enough.
    // FSs that implement permissions should override this.
    if (!this.mkdirs(f, absolutePermission)) {
      throw new IOException("mkdir of "+ f + " failed");
    }
  }

  /**
   * Opens an FSDataOutputStream at the indicated Path with write-progress
   * reporting. Same as create(), except fails if parent directory doesn't
   * already exist.
   * @param f the file name to open
   * @param overwrite if a file with this name already exists, then if true,
   * the file will be overwritten, and if false an error will be thrown.
   * @param bufferSize the size of the buffer to be used.
   * @param replication required block replication for the file.
   * @param blockSize block size
   * @param progress the progress reporter
   * @throws IOException IO failure
   * @see #setPermission(Path, FsPermission)
   * @return output stream.
   */

View on GitHub (pinned to 2add963021)

Solutions

1. Check what currently sits at f via fs.getFileStatus(f): if a file, delete it or pick another name
2. Verify write permission on the parent directory for the current user (and kinit for Kerberos clusters)
3. Prefer fs.mkdirs(f) after an explicit exists() check so the false path is distinguishable

Example fix

// before
fs.primitiveMkdir(f, FsPermission.getDefault(), true); // IOException: mkdir of f failed

// after
if (fs.exists(f) && !fs.getFileStatus(f).isDirectory()) {
  fs.delete(f, false); // stale file blocks the mkdir
}
boolean ok = fs.mkdirs(f);

if (fs.exists(f) && !fs.getFileStatus(f).isDirectory()) {
  fs.delete(f, false); // stale file blocks the mkdir
}
if (!fs.exists(f.getParent())) {
  fs.mkdirs(f.getParent());
}
boolean ok = fs.mkdirs(f);

When it happens

Trigger: mkdirs(f, permission) returning false: f already exists as a file, the effective user lacks write permission on the parent in the backing store, or a store-level error surfaced as a boolean false.

Common situations: Job rerun writing over an old file-shaped output; HDFS permissions for an unauthenticated/impersonated user under /user; object-store or custom filesystems mapping auth errors to false.

Related errors

• Mkdirs failed to create {} (exists={}, cwd={})
• File already exists: ${path}. Append or overwrite option mus
• Missing parent:${f}
• parent is not a dir
• rename destination ${dst} already exists.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/6453e8aa1c5acf32. Report an issue: GitHub.