apache/hadoop · warning · RenameFailedException

Destination is a non-empty directory

Error message

Destination is a non-empty directory

What it means

Source is a directory and the destination exists as a directory that is not provably empty (dstStatus.isEmptyDirectory() != Tristate.TRUE covers both non-empty and 'unknown', since marker-less directories cannot be proven empty). RenameFailedException 'Destination is a non-empty directory' is thrown with exit code false; rename() catches it, logs INFO, and returns false - matching POSIX rename(2) ENOTEMPTY/EEXIST semantics.

Source

Thrown at hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java:2467

      throw new RenameFailedException(src, dst,
          "source and dest refer to the same file or directory")
          .withExitCode(srcStatus.isFile());
    }

    S3AFileStatus dstStatus = null;
    try {
      dstStatus = innerGetFileStatus(dst, true, StatusProbeEnum.ALL);
      // if there is no destination entry, an exception is raised.
      // hence this code sequence can assume that there is something
      // at the end of the path; the only detail being what it is and
      // whether or not it can be the destination of the rename.
      if (srcStatus.isDirectory()) {
        if (dstStatus.isFile()) {
          throw new FileAlreadyExistsException(
              "Failed to rename " + src + " to " + dst
               +"; source is a directory and dest is a file");
        } else if (dstStatus.isEmptyDirectory() != Tristate.TRUE) {
          throw new RenameFailedException(src, dst,
              "Destination is a non-empty directory")
              .withExitCode(false);
        }
        // at this point the destination is an empty directory
      } else {
        // source is a file. The destination must be a directory,
        // empty or not
        if (dstStatus.isFile()) {
          throw new FileAlreadyExistsException(
              "Failed to rename " + src + " to " + dst
                  + "; destination file exists");
        }
      }

    } catch (FileNotFoundException e) {
      LOG.debug("rename: destination path {} not found", dst);
      // Parent must exist
      Path parent = dst.getParent();

View on GitHub (pinned to 2add963021)

Solutions

1. Clean the destination first: fs.delete(dst, true), then rename
2. Use a fresh destination path (timestamped) instead of merging
3. For merge semantics, copy children individually (FileUtil.copy, distcp) rather than rename

Example fix

// before: returns false when dst is non-empty
boolean ok = fs.rename(src, dst);

// after: explicit cleanup makes intent visible
if (fs.exists(dst)) {
  fs.delete(dst, true);
}
boolean ok = fs.rename(src, dst);

static boolean destinationAcceptsRename(FileSystem fs, Path dst) throws IOException {
  if (!fs.exists(dst)) return true;
  FileStatus st = fs.getFileStatus(dst);
  return st.isDirectory() && st instanceof S3AFileStatus
      && ((S3AFileStatus) st).isEmptyDirectory() == Tristate.TRUE;
}

When it happens

Trigger: rename(srcDir, dstDir) where dstDir contains objects or a KEEP marker, or its emptiness cannot be determined from the listing; large trees where directory markers are not authoritative.

Common situations: Attempting directory 'merge' via rename (not supported); destination not cleaned from a previous run; directory marker retention policies making emptiness indeterminate.

Related errors

• Rename cannot overwrite non empty destination directory {dst
• rename cannot overwrite non empty destination directory ${ds
• source is root directory
• dest is root directory
• source and dest refer to the same file or directory

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/3de9daa31059ab6c. Report an issue: GitHub.