apache/hadoop · error · PathIOException

{stageStatisticName}: mkdirs() returned false

Error message

{stageStatisticName}: mkdirs() returned false

What it means

Manifest committer stages create directories through operations.mkdirs() tracked under the OP_MKDIRS statistic. Java's FileSystem.mkdirs() returns false (rather than throwing) when creation fails; when escalateFailure is true the stage converts that false into PathIOException('<stageStatisticName>: mkdirs() returned false'). A false return usually means the target exists as a file, or the caller lacks permission to create it.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/committer/manifest/stages/AbstractJobOrTaskStage.java:496

        operations.deleteFile(path));
  }

  /**
   * Create a directory.
   * @param path path
   * @param escalateFailure escalate "false" to PathIOE
   * @return true if the directory was created/exists.
   * @throws IOException IO Failure.
   */
  public final boolean mkdirs(
      final Path path,
      final boolean escalateFailure)
      throws IOException {
    LOG.trace("{}: mkdirs('{}')", getName(), path);
    return trackDuration(getIOStatistics(), OP_MKDIRS, () -> {
      boolean success = operations.mkdirs(path);
      if (!success && escalateFailure) {
        throw new PathIOException(path.toUri().toString(),
            stageStatisticName + ": mkdirs() returned false");
      }
      return success;
    });

  }

  /**
   * List all directly files under a path.
   * Async implementations may under-report their durations.
   * @param path path
   * @return iterator over the results.
   * @throws IOException IO Failure.
   */
  protected final RemoteIterator<FileStatus> listStatusIterator(
      final Path path)
      throws IOException {
    LOG.trace("{}: listStatusIterator('{}')", getName(), path);

View on GitHub (pinned to 2add963021)

Solutions

1. Stat the failing path (printed in the PathIOException): if it is a file, delete or relocate it and rerun.
2. Fix ownership/permissions on the output tree so the running user can create directories.
3. Rerun into a fresh output directory to rule out stale state.
4. Check the nested filesystem diagnostics/logs around the OP_MKDIRS statistic for the underlying refusal.

// before running the job: every directory path the committer will create
// must be absent or already a directory, and its parent writable
FileSystem fs = outputDir.getFileSystem(conf);
if (fs.exists(outputDir) && !fs.getFileStatus(outputDir).isDirectory()) {
  throw new IOException("Output path is a file, not a directory: " + outputDir);
}
if (fs.exists(outputDir)
    && !fs.getFileStatus(outputDir).getPermission().getUserAction().implies(FsAction.WRITE)) {
  throw new IOException("Output dir not writable: " + outputDir);
}

When it happens

Trigger: Creating a stage directory (job/job-attempt/task attempt paths in the output tree) where: a FILE already exists at that exact path, the parent is not writable, or the filesystem is read-only/unavailable in a way that surfaces as a boolean failure.

Common situations: A leftover file occupying a directory path from an earlier failed run; output directories owned by another user; quota or permission changes between job submission and commit; a user or external process having written into the committer's namespace.

Related errors

• Mkdirs failed to create {}
• Mkdirs failed to create {} (exists={}, cwd={})
• Permission denied: user=%s, path="%s":%s:%s:%s%s
• {} doesn't support modifyAclEntries
• {} doesn't support removeAclEntries

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/1e2b4bd887be78e1. Report an issue: GitHub.