apache/hadoop · error · FileAlreadyExistsException

<path> already exists

Error message

<path> already exists

What it means

The pre-create probes found an existing object at the path and it is a file, but CreateFlag.OVERWRITE is not set, so create raises FileAlreadyExistsException '<path> already exists' instead of clobbering data. With overwrite=true the same probe result is only logged ('Overwriting file') and the write proceeds.

Source

Thrown at hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java:2172

    if (skipList) {
      probes.remove(StatusProbeEnum.List);
    }

    // if probes are required -request them and evaluate the result.
    if (!probes.isEmpty()) {
      try {

        // get the status or throw an FNFE.
        FileStatus status = innerGetFileStatus(path, false, probes);

        // if the thread reaches here, there is something at the path
        if (status.isDirectory()) {
          // path references a directory: automatic error
          throw new FileAlreadyExistsException(path + " is a directory");
        }
        if (!overwrite) {
          // path references a file and overwrite is disabled
          throw new FileAlreadyExistsException(path + " already exists");
        }
        LOG.debug("Overwriting file {}", path);
      } catch (FileNotFoundException e) {
        // this means there is nothing at the path; all good.
      }
    } else {
      LOG.debug("Skipping all probes with flags:"
              + " createPerf={}, magic={}, ccAvailable={}, cCreate={}, cEtag={}",
          createPerf, magic, ccAvailable, cCreate, cEtag);
    }
    instrumentation.fileCreated();
    final BlockOutputStreamStatistics outputStreamStatistics
        = statisticsContext.newOutputStreamStatistics();
    PutTracker putTracker =
        committerIntegration.createTracker(path, key, outputStreamStatistics);
    String destKey = putTracker.getDestKey();

    EnumSet<WriteObjectFlags> putFlags = options.writeObjectFlags();

View on GitHub (pinned to 2add963021)

Solutions

  1. Pass overwrite=true explicitly when replacing is intended: fs.create(path, true), or delete the object first
  2. For race-safe no-clobber creates use conditional writes (fs.s3a.create.conditional.enabled) or unique per-attempt names
  3. Clean stale output in job setup (e.g. delete the output directory before submission)

Example fix

// before: defaults to overwrite=false, throws on reruns
FSDataOutputStream out = fs.create(outputFile);

// after: intent is explicit
FSDataOutputStream out = fs.create(outputFile, true);
Defensive patterns

Strategy: try-catch

Validate before calling

// Race-aware no-clobber create
if (fs.exists(path)) {
  throw new IllegalStateException("output already published: " + path);
}
out = fs.create(path, false);

Try / catch

Catch FileAlreadyExistsException from create(); for single-writer flows decide overwrite upfront (create(path, true)); for concurrent writers treat it as 'someone else won' and skip or pick a unique name - do not blindly retry.

Prevention

When it happens

Trigger: fs.create(path) or fs.create(path, false) - the default no-overwrite form - when an object already exists at path; two writers racing to create the same output file.

Common situations: Job reruns without cleanup of previous output; MapReduce/Spark task retries writing the same part file; code ported from systems where create overwrites by default.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/5ca8f8ba8604fef9. Report an issue: GitHub.