apache/hadoop · error · IOException

rename cannot overwrite non empty destination directory ${ds

Error message

rename cannot overwrite non empty destination directory ${dst}

What it means

With Rename.OVERWRITE, the deprecated rename replaces the destination only via delete(dst, false), which cannot remove a populated directory. If listStatus(dst) is non-empty it throws IOException('rename cannot overwrite non empty destination directory dst'): overwrite is limited to files and empty directories.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java:1706

    try {
      dstStatus = getFileLinkStatus(dst);
    } catch (IOException e) {
      dstStatus = null;
    }
    if (dstStatus != null) {
      if (srcStatus.isDirectory() != dstStatus.isDirectory()) {
        throw new IOException("Source " + src + " Destination " + dst
            + " both should be either file or directory");
      }
      if (!overwrite) {
        throw new FileAlreadyExistsException("rename destination " + dst
            + " already exists.");
      }
      // Delete the destination that is a file or an empty directory
      if (dstStatus.isDirectory()) {
        FileStatus[] list = listStatus(dst);
        if (list != null && list.length != 0) {
          throw new IOException(
              "rename cannot overwrite non empty destination directory " + dst);
        }
      }
      delete(dst, false);
    } else {
      final Path parent = dst.getParent();
      final FileStatus parentStatus = getFileStatus(parent);
      if (parentStatus == null) {
        throw new FileNotFoundException("rename destination parent " + parent
            + " not found.");
      }
      if (!parentStatus.isDirectory()) {
        throw new ParentNotDirectoryException("rename destination parent " + parent
            + " is a file.");
      }
    }
    if (!rename(src, dst)) {
      throw new IOException("rename from " + src + " to " + dst + " failed.");

View on GitHub (pinned to 2add963021)

Solutions

Delete the destination tree explicitly first: fs.delete(dst, true), then rename
Or rename INTO the directory (dst = dstDir/srcName) instead of over it
Use per-run destination names and clean old runs separately

Example fix

// before
fs.rename(srcDir, dstDir, Rename.OVERWRITE); // dstDir non-empty

// after
if (fs.exists(dstDir) && fs.getFileStatus(dstDir).isDirectory()) {
  fs.delete(dstDir, true);
}
fs.rename(srcDir, dstDir);

Defensive patterns

Strategy: validation

Validate before calling

if (fs.exists(dst)) {
  FileStatus s = fs.getFileStatus(dst);
  boolean replaceable = !s.isDirectory() || fs.listStatus(dst).length == 0;
  if (!replaceable) {
    fs.delete(dst, true); // OVERWRITE can only replace files/empty dirs
  }
}
fs.rename(src, dst, Rename.OVERWRITE);

Prevention

Never assume recursive-overwrite semantics from Rename.OVERWRITE
Clean destination trees as an explicit pipeline step
Prefer renaming into a directory over replacing a directory

When it happens

Trigger: rename(dir -> existing non-empty dir, OVERWRITE), e.g. publishing a directory of part-files onto a previous run's populated output directory.

Common situations: Users expecting 'mv'-style recursive replacement; staging/commit code renaming whole result trees onto existing trees.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/11740904234102ed. Report an issue: GitHub.