apache/hadoop · error · FileAlreadyExistsException

{operation}: path {path} already exists and has status {stat

Error message

{operation}: path {path} already exists and has status {status}

What it means

createNewDirectory() implements create-if-absent semantics: it probes getFileStatus first, and if the probe succeeds (no FileNotFoundException) the path already exists, so it throws FileAlreadyExistsException('<operation>: path <path> already exists and has status <status>'). Manifest-committer job/job-attempt directories are expected to be new; an existing one means stale state or a concurrent committer.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/committer/manifest/stages/AbstractJobOrTaskStage.java:578

   * mkdirs() failed.
   * @param operation operation for error reporting.
   * @param path path path to create.
   * @return the path.
   * @throws IOException failure
   * @throws PathIOException mkdirs failed.
   * @throws FileAlreadyExistsException destination exists.
   */
  protected final Path createNewDirectory(
      final String operation,
      final Path path) throws IOException {
    LOG.trace("{}: {} createNewDirectory('{}')", getName(), operation, path);
    requireNonNull(path,
        () -> String.format("%s: Null path for operation %s", getName(), operation));
    // check for dir existence before trying to create.
    try {
      final FileStatus status = getFileStatus(path);
      // no exception, so the path exists.
      throw new FileAlreadyExistsException(operation
          + ": path " + path
          + " already exists and has status " + status);
    } catch (FileNotFoundException e) {
      // the path does not exist, so create it.
      mkdirs(path, true);
      return path;
    }
  }

  /**
   * Assert that a path is a directory which must exist.
   * @param operation operation for error reporting.
   * @param path path path to create.
   * @return the path
   * @throws IOException failure
   * @throws PathIOException mkdirs failed.
   * @throws FileAlreadyExistsException destination exists.
   */

View on GitHub (pinned to 2add963021)

Solutions

1. Clean stale committer state: remove the manifest committer's intermediate tree under the output path (or the whole output dir if it is disposable) and rerun.
2. Write to a fresh, job-unique output directory per run.
3. Ensure only one job at a time commits into a given output tree.
4. Read the status in the message - it tells you whether you are colliding with a file or an earlier attempt's directory.

Example fix

// before: rerunning a failed job into the same output
Path out = new Path("abfs://container/data/job-out"); // contains stale attempt dirs

// after: clean or version the output before resubmitting
FileSystem fs = out.getFileSystem(conf);
if (fs.exists(out)) { fs.delete(out, true); } // only if disposable
// or: Path out = new Path("abfs://container/data/job-out-" + runId);

// before resubmitting into a reused output path, clear stale committer state
FileSystem fs = outputDir.getFileSystem(conf);
if (fs.exists(outputDir)) {
  // whole-tree delete if disposable; otherwise remove the committer's intermediate dirs only
  fs.delete(outputDir, true);
}

try {
  job.waitForCompletion(true);
} catch (FileAlreadyExistsException e) {
  // stale attempt dirs: clean the output tree (or use a new output dir) and resubmit
}

When it happens

Trigger: A manifest committer stage creating a directory (job attempt dir etc.) that already exists: leftover from a prior failed attempt with the same job id, a concurrent job committing into the same output tree, or manual recreation of committer directories.

Common situations: Resubmitting a failed job with the same job id into the same output without cleanup; two applications sharing an output path; operators 'pre-creating' committer directory structures; repeated AM retries racing the commit.

Related errors

• {} already exists
• Not a directory: {}
• Path is a file: {}
• Can't make directory for path '%s' since it is a file.
• File: %s already exists

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/c4d80bf34e0fde7d. Report an issue: GitHub.