apache/hadoop · error · PathExistsException

E_DEST_EXISTS

E_DEST_EXISTS

Error message

output path is not a directory: Destination path exists and committer conflict resolution mode is "fail"

What it means

PathExistsException from DirectoryStagingCommitter.setupJob when the destination path already exists but is NOT a directory. setupJob runs getFileStatus on the output path first and fails fast for every conflict-resolution mode (fail, append, replace) when the existing entry is a file, because a file at the output root cannot receive task output. The message text bundles the E_DEST_EXISTS constant, which makes it read like a conflict-mode complaint even though the deciding condition here is '!status.isDirectory()'.

Source

Thrown at hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/staging/DirectoryStagingCommitter.java:78

  @Override
  public String getName() {
    return NAME;
  }

  @Override
  public void setupJob(JobContext context) throws IOException {
    Path outputPath = getOutputPath();
    FileSystem fs = getDestFS();
    ConflictResolution conflictResolution = getConflictResolutionMode(
        context, fs.getConf());
    LOG.info("Conflict Resolution mode is {}", conflictResolution);
    try {
      final FileStatus status = fs.getFileStatus(outputPath);

      // if it is not a directory, fail fast for all conflict options.
      if (!status.isDirectory()) {
        throw new PathExistsException(outputPath.toString(),
            "output path is not a directory: "
                + InternalCommitterConstants.E_DEST_EXISTS);
      }
      switch(conflictResolution) {
      case FAIL:
        throw failDestinationExists(outputPath,
            "Setting job as " + getRole());
      case APPEND:
      case REPLACE:
        LOG.debug("Destination directory exists; conflict policy permits this");
      }
    } catch (FileNotFoundException ignored) {
      // there is no destination path, hence, no conflict.
    }
    // make the parent directory, which also triggers a recursive directory
    // creation operation
    super.setupJob(context);
  }

View on GitHub (pinned to 2add963021)

Solutions

Delete the existing file at the destination path (hadoop fs -rm s3a://bucket/output) or choose a fresh output path
If the destination is an existing directory and you still want strict semantics, review fs.s3a.committer.staging.conflict-mode (values: fail, append, replace; default append) -- but note no mode tolerates a file at the destination
Add a job-level preflight check that lists the output path and fails with a clear message before the cluster is allocated

Example fix

# before: 'output' exists as a file from a prior run
hadoop jar job.jar -Dmapreduce.output.fileoutputformat.outputdir=s3a://bucket/output ...

# after: remove the stale object first
hadoop fs -rm s3a://bucket/output
hadoop jar job.jar -Dmapreduce.output.fileoutputformat.outputdir=s3a://bucket/output ...

Defensive patterns

Strategy: validation

Validate before calling

try {
  FileStatus st = fs.getFileStatus(outputPath);
  if (!st.isDirectory()) {
    throw new IOException("Output path exists as a FILE; remove it or choose a new path: "
        + outputPath);
  }
} catch (FileNotFoundException ok) {
  // absent is fine
}

Try / catch

try {
  committer.setupJob(context);
} catch (PathExistsException e) {
  // destination exists as a file: no conflict mode rescues this; remove/rename it and resubmit
  LOG.error("Output destination occupied by a file: {}", outputPath, e);
}

Prevention

Run a preflight 'does the output path exist and is it a file' check in the job driver
Use unique per-run output paths (timestamps/run ids) in shared buckets
Do not create marker objects at intended output-directory locations

When it happens

Trigger: Starting a job whose mapreduce output path resolves to an existing S3 object (file), e.g. s3a://bucket/output where 'output' is a zero-byte marker object or a previously written single file.

Common situations: A prior run or manual upload left a file at the exact output path; console tools or other applications create an empty object as a path marker; a prior job with a single output file wrote directly to the output root.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/6858499b1d9a7d1e. Report an issue: GitHub.