apache/hadoop · error · IOException

Source {src} and destination {dst} must both be directories

Error message

Source {src} and destination {dst} must both be directories

What it means

During overwrite rename, if the destination exists but source and destination disagree on isDirectory(), AbstractFileSystem.renameInternal throws this plain IOException: you may only overwrite a file with a file or a directory with a directory. Note the source comment: renaming a file to a symlink (or vice versa) is allowed because the link's own status is not a directory, so a directory-vs-symlink pair also lands here.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/AbstractFileSystem.java:872

    FileStatus dstStatus;
    try {
      dstStatus = getFileLinkStatus(dst);
    } catch (IOException e) {
      dstStatus = null;
    }
    if (dstStatus != null) {
      if (dst.equals(src)) {
        throw new FileAlreadyExistsException(
            "The source "+src+" and destination "+dst+" are the same");
      }
      if (srcStatus.isSymlink() && dst.equals(srcStatus.getSymlink())) {
        throw new FileAlreadyExistsException(
            "Cannot rename symlink "+src+" to its target "+dst);
      }
      // It's OK to rename a file to a symlink and vice versa
      if (srcStatus.isDirectory() != dstStatus.isDirectory()) {
        throw new IOException("Source " + src + " and destination " + dst
            + " must both be directories");
      }
      if (!overwrite) {
        throw new FileAlreadyExistsException("Rename destination " + dst
            + " already exists.");
      }
      // Delete the destination that is a file or an empty directory
      if (dstStatus.isDirectory()) {
        RemoteIterator<FileStatus> list = listStatusIterator(dst);
        if (list != null && list.hasNext()) {
          throw new IOException(
              "Rename cannot overwrite non empty destination directory " + dst);
        }
      }
      delete(dst, false);
    } else {
      final Path parent = dst.getParent();
      final FileStatus parentStatus = getFileStatus(parent);

View on GitHub (pinned to 2add963021)

Solutions

1. Delete the mismatched destination first (fc.delete(dst, true)) and then perform the overwrite rename
2. Choose a fresh destination path so a type collision is impossible
3. Normalize the pipeline so a given output path is always the same type across runs

Example fix

// before
fc.rename(srcDir, dst, Options.Rename.OVERWRITE); // dst is a file -> IOException

// after
if (fc.util().exists(dst)
    && fc.getFileStatus(src).isDirectory() != fc.getFileStatus(dst).isDirectory()) {
  fc.delete(dst, true);
}
fc.rename(srcDir, dst, Options.Rename.OVERWRITE);

if (fc.util().exists(dst)
    && fc.getFileStatus(src).isDirectory()
        != fc.getFileStatus(dst).isDirectory()) {
  fc.delete(dst, true); // or choose a different dst
}

catch (IOException e) { if (e.getMessage().startsWith("Source ")) { /* type conflict: delete dst or retarget */ } else throw e; }

When it happens

Trigger: fc.rename(srcDir, dstFile, Rename.OVERWRITE) or fc.rename(srcFile, dstDir, Rename.OVERWRITE) where dst exists; also renaming a directory onto an existing symlink (the symlink's own FileStatus is not a directory).

Common situations: Job reruns where a previous run left a _SUCCESS marker file at a path now expected to be a directory (or the reverse); committers switching output layout between file and directory; leftover artifacts from an older pipeline schema occupying the destination name.

Related errors

• Rename cannot overwrite non empty destination directory {dst
• Cannot rename symlink {src} to its target {dst}
• Rename destination {dst} already exists.
• Rename destination parent {parent} is a file.
• new destination is an existed file

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/20062cd53ffcf5db. Report an issue: GitHub.