apache/hadoop · error · IOException

Failed to create directory: {}

Error message

Failed to create directory: {}

What it means

IOException ('Failed to create directory') raised when the internal unprotected mkdir could not add the new inode - fsd.addLastINode returned null, which happens when the namespace changed under the caller between path resolution and insertion. It is the NameNode's sentinel for a lost creation race (typically another client just created the same component); quota failures throw QuotaExceededException instead and a file in the way throws FileAlreadyExistsException.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirMkdirOp.java:83

        if (!createParent) {
          fsd.verifyParentDir(iip);
        }

        // validate that we have enough inodes. This is, at best, a
        // heuristic because the mkdirs() operation might need to
        // create multiple inodes.
        fsn.checkFsObjectLimit();

        // Ensure that the user can traversal the path by adding implicit
        // u+wx permission to all ancestor directories.
        INodesInPath existing =
            createParentDirectories(fsd, iip, permissions, false);
        if (existing != null) {
          existing = createSingleDirectory(
              fsd, existing, iip.getLastLocalName(), permissions);
        }
        if (existing == null) {
          throw new IOException("Failed to create directory: " + src);
        }
        iip = existing;
      }
      return fsd.getAuditFileInfo(iip);
    } finally {
      fsd.writeUnlock();
    }
  }

  /**
   * For a given absolute path, create all ancestors as directories along the
   * path. All ancestors inherit their parent's permission plus an implicit
   * u+wx permission. This is used by create() and addSymlink() for
   * implicitly creating all directories along the path.
   *
   * For example, path="/foo/bar/spam", "/foo" is an existing directory,
   * "/foo/bar" is not existing yet, the function will create directory bar.
   *

View on GitHub (pinned to 2add963021)

Solutions

1. Re-check the path: `hdfs dfs -test -d <path>` - if it now exists, the goal is met; continue.
2. Retry mkdirs; it is idempotent and a small bounded retry loop almost always resolves the race.
3. Create shared root directories once at deploy time so parallel jobs only mkdir distinct children.

Example fix

// before
fs.mkdirs(deepPath); // IOException: Failed to create directory (lost race)

// after
static void mkdirsRetry(FileSystem fs, Path p, int attempts) throws IOException {
  IOException last = null;
  for (int i = 0; i < attempts; i++) {
    try { fs.mkdirs(p); return; }
    catch (IOException e) {
      last = e;
      if (fs.exists(p) && fs.getFileStatus(p).isDirectory()) return; // race lost but dir exists
    }
  }
  throw last;
}

static void mkdirsRetry(FileSystem fs, Path p, int attempts) throws IOException {
  IOException last = null;
  for (int i = 0; i < attempts; i++) {
    try {
      fs.mkdirs(p);
      return;
    } catch (IOException e) {
      last = e;
      // lost race but the outcome exists: done
      if (fs.exists(p) && fs.getFileStatus(p).isDirectory()) return;
    }
  }
  throw last;
}

When it happens

Trigger: Two clients running `hdfs dfs -mkdir -p` on the same deep path concurrently; many parallel task attempts initializing a shared staging directory; a burst of retries right after a failed job submit.

Common situations: Highly concurrent job launchers (Oozie, Spark, workflow templates) creating common parent directories; parallelized init scripts across nodes; shared warehouse bootstrap code.

Related errors

• No such file or directory
• Input/output error
• Another {name} is running.
• Commit or complete block {commitBlock}, whereas it is under
• Recovery block {b} where it is not under construction.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/e570bea4ea1e7d20. Report an issue: GitHub.