What causes "{path} already exists" in apache/hadoop?

Pass overwrite=true when the intent is to replace the object: fs.create(path, true)

apache/hadoop · error · FileAlreadyExistsException

{path} already exists

Error message

{path} already exists

What it means

Thrown by AliyunOSSFileSystem.create(): the target path exists as a file and the caller did not set the overwrite flag, so the connector raises FileAlreadyExistsException(path + " already exists") to protect existing data.

Source

Thrown at hadoop-tools/hadoop-aliyun/src/main/java/org/apache/hadoop/fs/aliyun/oss/AliyunOSSFileSystem.java:132

  @Override
  public FSDataOutputStream create(Path path, FsPermission permission,
      boolean overwrite, int bufferSize, short replication, long blockSize,
      Progressable progress) throws IOException {
    String key = pathToKey(path);
    FileStatus status = null;

    try {
      // get the status or throw a FNFE
      status = getFileStatus(path);

      // if the thread reaches here, there is something at the path
      if (status.isDirectory()) {
        // path references a directory
        throw new FileAlreadyExistsException(path + " is a directory");
      }
      if (!overwrite) {
        // path references a file and overwrite is disabled
        throw new FileAlreadyExistsException(path + " already exists");
      }
      LOG.debug("Overwriting file {}", path);
    } catch (FileNotFoundException e) {
      // this means the file is not found
    }

    return new FSDataOutputStream(
        new AliyunOSSBlockOutputStream(getConf(),
            store,
            key,
            uploadPartSize,
            blockFactory,
            blockOutputStreamStatistics,
            new SemaphoredDelegatingExecutor(boundedThreadPool,
                blockOutputActiveBlocks, true)), statistics);
  }

  /**

View on GitHub (pinned to 2add963021)

Solutions

Pass overwrite=true when the intent is to replace the object: fs.create(path, true)
If overwrite is wrong (idempotent pipelines), delete or move the existing object before the run
Make output file names unique per run/attempt (append attempt id or timestamp) so reruns never collide

Example fix

// before
fs.create(path, /*overwrite*/ false);

// after
fs.create(path, /*overwrite*/ true);

Defensive patterns

Strategy: validation

Validate before calling

boolean overwrite = /* rerun policy */ true;
if (!overwrite && fs.exists(path)) {
  throw new IOException("Output already exists: " + path);
}
fs.create(path, overwrite);

Try / catch

catch (FileAlreadyExistsException e) { if ("already exists".in(e.getMessage())) { /* decide: delete stale object or rerun with overwrite */ } throw e; }

Prevention

Decide the overwrite policy explicitly at every create() call site
Use per-run unique file names for idempotent pipelines instead of relying on overwrite

When it happens

Trigger: Calling create(path, /*overwrite*/ false, ...) when an object already exists at that key; re-running a job whose previous output object was not cleaned (e.g., failed commit, manually copied file).

Common situations: Re-executed Spark/MapReduce jobs writing to deterministic file names; FileOutputCommitter leftovers when a rerun uses the same output file name; shared output locations across teams where another writer created the object first.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/6f130cc9c28a221c. Report an issue: GitHub.