apache/hadoop · error · ParentNotDirectoryException

Rename destination parent

Error message

Rename destination parent {parent} is a file.

What it means

When the destination does not exist, renameInternal validates its parent: if getFileStatus(dst.getParent()) resolves to a regular file, the destination cannot be created beneath it and ParentNotDirectoryException is thrown before renameInternal runs. This is the same ENOTDIR-style failure the HDFS NameNode reports, surfaced client-side by FileContext.

Solutions

1. Remove or relocate the conflicting parent file, then fc.mkdir(parent, permission, true) before renaming
2. Adjust destination naming so every parent component is always a directory
3. Validate dst.getParent() with getFileStatus at job startup and fail fast with a clear message

Example fix

// before
fc.rename(src, new Path("/out/part-0/data")); // /out/part-0 is a file

// after
Path parent = dst.getParent();
if (fc.util().exists(parent) && fc.getFileStatus(parent).isFile()) {
  fc.delete(parent, false);
  fc.mkdir(parent, FsPermission.getDirDefault(), true);
}
fc.rename(src, dst);

Path parent = dst.getParent();
if (fc.util().exists(parent)) {
  FileStatus ps = fc.getFileStatus(parent);
  if (!ps.isDirectory()) {
    throw new IllegalStateException("Rename parent is a file: " + parent);
  }
} else {
  fc.mkdir(parent, FsPermission.getDirDefault(), true);
}

catch (ParentNotDirectoryException e) { /* remove/relocate the conflicting parent file, mkdir parent, retry */ }

When it happens

Trigger: fc.rename(f, new Path("/out/part-0/data")) when /out/part-0 is an existing file; dynamically built paths (date/part keys) whose parent component collides with a file written by an earlier flat layout.

Common situations: Output layouts that append extra path components after an upstream schema change; a previous run writing a file exactly where the next run expects a directory; committer working-dir misconfiguration in MapReduce/Spark jobs.

Related errors

• Cannot rename symlink
• Rename cannot overwrite non empty destination directory
• Rename destination already exists.
• Source and destination must both be directories
• File system does not support symlinks

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/aec790598879d976. Report an issue: GitHub.

Appendix: source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/AbstractFileSystem.java:892

      }
      if (!overwrite) {
        throw new FileAlreadyExistsException("Rename destination " + dst
            + " already exists.");
      }
      // Delete the destination that is a file or an empty directory
      if (dstStatus.isDirectory()) {
        RemoteIterator<FileStatus> list = listStatusIterator(dst);
        if (list != null && list.hasNext()) {
          throw new IOException(
              "Rename cannot overwrite non empty destination directory " + dst);
        }
      }
      delete(dst, false);
    } else {
      final Path parent = dst.getParent();
      final FileStatus parentStatus = getFileStatus(parent);
      if (parentStatus.isFile()) {
        throw new ParentNotDirectoryException("Rename destination parent "
            + parent + " is a file.");
      }
    }
    renameInternal(src, dst);
  }
  
  /**
   * Returns true if the file system supports symlinks, false otherwise.
   * @return true if filesystem supports symlinks
   */
  public boolean supportsSymlinks() {
    return false;
  }
  
  /**
   * The specification of this method matches that of  
   * {@link FileContext#createSymlink(Path, Path, boolean)};
   *

View on GitHub (pinned to 2add963021)