apache/hadoop · error · FileAlreadyExistsException

Failed to rename <src> to <dst>; destination file exists

Error message

Failed to rename <src> to <dst>; destination file exists

What it means

In initiateRename(), the source is a file and the destination also exists as a file. Hadoop's rename contract never overwrites an existing destination file, so FileAlreadyExistsException 'Failed to rename <src> to <dst>; destination file exists' is thrown and propagates out of rename() (it is not converted to a false return like RenameFailedException).

Source

Thrown at hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java:2476

      // hence this code sequence can assume that there is something
      // at the end of the path; the only detail being what it is and
      // whether or not it can be the destination of the rename.
      if (srcStatus.isDirectory()) {
        if (dstStatus.isFile()) {
          throw new FileAlreadyExistsException(
              "Failed to rename " + src + " to " + dst
               +"; source is a directory and dest is a file");
        } else if (dstStatus.isEmptyDirectory() != Tristate.TRUE) {
          throw new RenameFailedException(src, dst,
              "Destination is a non-empty directory")
              .withExitCode(false);
        }
        // at this point the destination is an empty directory
      } else {
        // source is a file. The destination must be a directory,
        // empty or not
        if (dstStatus.isFile()) {
          throw new FileAlreadyExistsException(
              "Failed to rename " + src + " to " + dst
                  + "; destination file exists");
        }
      }

    } catch (FileNotFoundException e) {
      LOG.debug("rename: destination path {} not found", dst);
      // Parent must exist
      Path parent = dst.getParent();
      if (!pathToKey(parent).isEmpty()
          && !parent.equals(src.getParent())) {
        try {
          // make sure parent isn't a file.
          // don't look for parent being a dir as there is a risk
          // of a race between dest dir cleanup and rename in different
          // threads.
          S3AFileStatus dstParentStatus = innerGetFileStatus(parent,
              false, StatusProbeEnum.FILE);

View on GitHub (pinned to 2add963021)

Solutions

1. Delete the destination object (or the whole output tree in job cleanup) before renaming
2. Publish via unique staging names plus an atomic final rename, or conditional create
3. Pre-check fs.exists(dst) and decide explicitly whether to replace

Example fix

// before: throws if dst object exists
fs.rename(staged, finalPath);

// after: remove stale destination deliberately
if (fs.exists(finalPath)) {
  fs.delete(finalPath, false);
}
fs.rename(staged, finalPath);

if (fs.exists(dst)) {
  fs.delete(dst, false);
}
boolean ok = fs.rename(src, dst);

Catch FileAlreadyExistsException from rename() in publish paths and treat it as a duplicate-publish signal: either the previous run's output is stale (delete it deliberately) or a concurrent publisher won the race (skip).

When it happens

Trigger: rename(file, file) where an object already exists at dst; two writers renaming different sources onto the same published name.

Common situations: Output already committed by a previous run; concurrent tasks racing to publish the same final path; publishing logic without unique staging names.

Related errors

• Failed to rename <src> to <dst>; source is a directory and d
• File: %s already exists
• Failed to rename since the dest path %s already exists.
• The source {src} and destination {dst} are the same
• The source {} and destination {} are the same

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/9e8f46fc104b1a07. Report an issue: GitHub.