apache/hadoop · error · FileAlreadyExistsException

Failed to rename <src> to <dst>; source is a directory and d

Error message

Failed to rename <src> to <dst>; source is a directory and dest is a file

What it means

In initiateRename(), after both statuses resolved successfully: srcStatus.isDirectory() and dstStatus.isFile(). A directory tree cannot be moved onto a single object, so FileAlreadyExistsException 'Failed to rename <src> to <dst>; source is a directory and dest is a file' is thrown. Unlike RenameFailedException this is not caught by rename(), so it propagates to the caller as an IOException.

Source

Thrown at hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java:2463

    if (srcKey.equals(dstKey)) {
      LOG.debug("rename: src and dest refer to the same file or directory: {}",
          dst);
      throw new RenameFailedException(src, dst,
          "source and dest refer to the same file or directory")
          .withExitCode(srcStatus.isFile());
    }

    S3AFileStatus dstStatus = null;
    try {
      dstStatus = innerGetFileStatus(dst, true, StatusProbeEnum.ALL);
      // if there is no destination entry, an exception is raised.
      // hence this code sequence can assume that there is something
      // at the end of the path; the only detail being what it is and
      // whether or not it can be the destination of the rename.
      if (srcStatus.isDirectory()) {
        if (dstStatus.isFile()) {
          throw new FileAlreadyExistsException(
              "Failed to rename " + src + " to " + dst
               +"; source is a directory and dest is a file");
        } else if (dstStatus.isEmptyDirectory() != Tristate.TRUE) {
          throw new RenameFailedException(src, dst,
              "Destination is a non-empty directory")
              .withExitCode(false);
        }
        // at this point the destination is an empty directory
      } else {
        // source is a file. The destination must be a directory,
        // empty or not
        if (dstStatus.isFile()) {
          throw new FileAlreadyExistsException(
              "Failed to rename " + src + " to " + dst
                  + "; destination file exists");
        }
      }

View on GitHub (pinned to 2add963021)

Solutions

1. Delete or move the destination file, then retry the rename
2. Choose a destination that does not exist or is an empty directory
3. Pre-check fs.getFileStatus(dst) before launching expensive work and fail early with context

Example fix

// before
fs.rename(srcDir, dst);

// after: clear a stale destination file first
if (fs.exists(dst) && fs.getFileStatus(dst).isFile()) {
  fs.delete(dst, false);
}
fs.rename(srcDir, dst);

if (fs.exists(dst) && fs.getFileStatus(dst).isFile()) {
  throw new IOException("destination exists as a file: " + dst);
}
fs.rename(src, dst);

Catch FileAlreadyExistsException from rename(): it escapes rename() (unlike RenameFailedException). Decide replace-vs-abort based on your publishing contract before deleting the destination file.

When it happens

Trigger: rename(dir, existingFile); destination previously written as a file while the source path later became a directory; concurrent jobs writing a file where another is renaming a directory.

Common situations: Schema evolution where yesterday's file path is today's partition directory; pipeline reruns with restructured output; recovery flows renaming directories over stale marker files.

Related errors

• Failed to rename <src> to <dst>; destination file exists
• File: %s already exists
• Failed to rename since the dest path %s already exists.
• The source {src} and destination {dst} are the same
• Source {src} and destination {dst} must both be directories

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/1784eda172ba67bc. Report an issue: GitHub.