apache/hadoop · error · IOException

Rename cannot overwrite non empty destination directory {dst

Error message

Rename cannot overwrite non empty destination directory {dst}

What it means

With Rename.OVERWRITE, Hadoop will only replace a destination directory that is empty: rename is an atomic replace, not a merge. renameInternal probes dst with listStatusIterator and throws this IOException as soon as one entry is returned, mirroring the HDFS NameNode's own non-empty-directory check.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/AbstractFileSystem.java:883

      }
      if (srcStatus.isSymlink() && dst.equals(srcStatus.getSymlink())) {
        throw new FileAlreadyExistsException(
            "Cannot rename symlink "+src+" to its target "+dst);
      }
      // It's OK to rename a file to a symlink and vice versa
      if (srcStatus.isDirectory() != dstStatus.isDirectory()) {
        throw new IOException("Source " + src + " and destination " + dst
            + " must both be directories");
      }
      if (!overwrite) {
        throw new FileAlreadyExistsException("Rename destination " + dst
            + " already exists.");
      }
      // Delete the destination that is a file or an empty directory
      if (dstStatus.isDirectory()) {
        RemoteIterator<FileStatus> list = listStatusIterator(dst);
        if (list != null && list.hasNext()) {
          throw new IOException(
              "Rename cannot overwrite non empty destination directory " + dst);
        }
      }
      delete(dst, false);
    } else {
      final Path parent = dst.getParent();
      final FileStatus parentStatus = getFileStatus(parent);
      if (parentStatus.isFile()) {
        throw new ParentNotDirectoryException("Rename destination parent "
            + parent + " is a file.");
      }
    }
    renameInternal(src, dst);
  }
  
  /**
   * Returns true if the file system supports symlinks, false otherwise.
   * @return true if filesystem supports symlinks

View on GitHub (pinned to 2add963021)

Solutions

1. Delete the destination tree first: fc.delete(dst, true), then perform the overwrite rename
2. Merge contents yourself (copy/move old entries aside) before overwriting
3. Use versioned destination names plus a pointer/alias switch instead of in-place overwrite

Example fix

// before
fc.rename(srcDir, dstDir, Options.Rename.OVERWRITE); // dstDir non-empty -> IOException

// after
if (fc.util().exists(dstDir)) {
  fc.delete(dstDir, true); // only if discarding old contents is intended
}
fc.rename(srcDir, dstDir, Options.Rename.OVERWRITE);

if (fc.util().exists(dst)) {
  FileStatus[] entries = fc.util().listStatus(dst);
  if (entries.length > 0) {
    fc.delete(dst, true); // only when discarding old contents is intended
  }
}

catch (IOException e) { if (e.getMessage().contains("non empty destination")) { /* delete dst recursively, then retry once */ } else throw e; }

When it happens

Trigger: fc.rename(srcDir, dstDir, Rename.OVERWRITE) where dstDir contains files or subdirectories; sync tools that assume rsync-style merge semantics; directories that look empty but hold hidden entries (.crc, _SUCCESS, .tmp part files).

Common situations: Atomic output committers publishing into a partition directory that still contains previous files; recovery after a partially committed earlier run; republishing dated partition paths.

Related errors

• Source {src} and destination {dst} must both be directories
• Cannot rename symlink {src} to its target {dst}
• Rename destination {dst} already exists.
• Rename destination parent {parent} is a file.
• new destination is an existed file

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/bbde0ad8121896b8. Report an issue: GitHub.