apache/hadoop · error · PathIOException

{operation}: Path is not a directory; its status is :{status

Error message

{operation}: Path is not a directory; its status is :{status}

What it means

directoryMustExist() asserts a required path exists AND is a directory; when getFileStatus succeeds but status.isDirectory() is false it throws PathIOException('<operation>: Path is not a directory; its status is :<status>'). Something the committer expects to traverse as a directory (e.g. task-attempt or manifest directories in the output tree) has been replaced by a regular file.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/committer/manifest/stages/AbstractJobOrTaskStage.java:602

      return path;
    }
  }

  /**
   * Assert that a path is a directory which must exist.
   * @param operation operation for error reporting.
   * @param path path path to create.
   * @return the path
   * @throws IOException failure
   * @throws PathIOException mkdirs failed.
   * @throws FileAlreadyExistsException destination exists.
   */
  protected final Path directoryMustExist(
      final String operation,
      final Path path) throws IOException {
    final FileStatus status = getFileStatus(path);
    if (!status.isDirectory()) {
      throw new PathIOException(path.toString(),
          operation
              + ": Path is not a directory; its status is :" + status);
    }
    return path;
  }

  /**
   * Save a task manifest or summary. This will be done by
   * writing to a temp path and then renaming.
   * If the destination path exists: Delete it.
   * This will retry so that a rename failure from abfs load or IO errors
   * will not fail the task.
   * @param manifestData the manifest/success file
   * @param tempPath temp path for the initial save
   * @param finalPath final path for rename.
   * @return the manifest saved.
   * @throws IOException failure to rename after retries.
   */

View on GitHub (pinned to 2add963021)

Solutions

1. Stat the path shown in the message; delete or relocate the file that occupies the directory location.
2. Keep external writers (markers, _SUCCESS-style files, downstream readers) out of committer-managed subtrees.
3. Rerun the job into a clean output directory if the tree's state is uncertain.
4. If this recurs, audit what process creates files at that path (check file owner/timestamps in the printed status).

// sanity-check structural paths before the job: required dirs must be dirs
for (Path p : requiredDirs) {
  if (fs.exists(p) && !fs.getFileStatus(p).isDirectory()) {
    throw new IOException("Expected directory but found file at " + p);
  }
}

static boolean isDirectoryOrAbsent(FileSystem fs, Path p) throws IOException {
  FileStatus st = fs.exists(p) ? fs.getFileStatus(p) : null;
  return st == null || st.isDirectory();
}

When it happens

Trigger: A stage resolving a structural path (manifest/task/job attempt directories) that exists as a file: a user or process created a file at that location, or a partial cleanup converted a directory path into a file.

Common situations: Operators or pipelines writing marker/success files into committer-managed directories; output trees shared with other writers; earlier crashed runs leaving a file where a directory belongs.

Related errors

• Mkdirs failed to create {} (exists={}, cwd={})
• Failed to delete {}
• {stageStatisticName}: mkdirs() returned false
• Expected a file renamed from {sourcePath}; found {destStatus
• Can't replace {target}. Target is {targetType}, Source is {s

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/f006f247bec29b36. Report an issue: GitHub.