apache/hadoop · warning · RenameFailedException

destination parent is not a directory

Error message

destination parent is not a directory

What it means

When the destination does not exist (the FileNotFoundException branch), initiateRename() checks dst's parent: if the parent resolves to something that is not a directory (i.e. an object), RenameFailedException 'destination parent is not a directory' is thrown with the default exit code, so rename() logs it at INFO and returns false. A missing parent is explicitly fine - S3A creates intermediate directories implicitly; the source notes the dir/file race only affects marker interpretation.

Source

Thrown at hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java:2498

      }

    } catch (FileNotFoundException e) {
      LOG.debug("rename: destination path {} not found", dst);
      // Parent must exist
      Path parent = dst.getParent();
      if (!pathToKey(parent).isEmpty()
          && !parent.equals(src.getParent())) {
        try {
          // make sure parent isn't a file.
          // don't look for parent being a dir as there is a risk
          // of a race between dest dir cleanup and rename in different
          // threads.
          S3AFileStatus dstParentStatus = innerGetFileStatus(parent,
              false, StatusProbeEnum.FILE);
          // if this doesn't raise an exception then
          // the parent is a file or a dir.
          if (!dstParentStatus.isDirectory()) {
            throw new RenameFailedException(src, dst,
                "destination parent is not a directory");
          }
        } catch (FileNotFoundException expected) {
          // nothing was found. Don't worry about it;
          // expect rename to implicitly create the parent dir
        }
      }
    }
    return Pair.of(srcStatus, dstStatus);
  }

  /**
   * The inner rename operation. See {@link #rename(Path, Path)} for
   * the description of the operation.
   * This operation throws an exception on any failure which needs to be
   * reported and downgraded to a failure.
   * Retries: retry translated, assuming all operations it is called do
   * so. For safely, consider catch and handle SdkException

View on GitHub (pinned to 2add963021)

Solutions

1. Delete the object occupying the ancestor path; the rename then recreates the directory chain
2. Correct the destination path construction
3. Validate the parent chain with fs.getFileStatus(dst.getParent()) before renaming

Example fix

// before: fails (returns false) if a/b exists as a file
fs.rename(src, new Path("a/b/c"));

// after: clear file-vs-directory conflicts in the ancestor chain
Path parent = dst.getParent();
while (parent != null && !parent.isRoot()) {
  if (fs.exists(parent) && fs.getFileStatus(parent).isFile()) {
    fs.delete(parent, false);
  }
  parent = parent.getParent();
}
fs.rename(src, dst);

static void ensureParentIsDirectory(FileSystem fs, Path dst) throws IOException {
  Path parent = dst.getParent();
  if (parent == null || parent.isRoot()) return;
  if (fs.exists(parent) && !fs.getFileStatus(parent).isDirectory()) {
    throw new IOException("ancestor of destination is a file: " + parent);
  }
}

When it happens

Trigger: rename(src, dst) where some ancestor of dst is an existing object, e.g. dst = a/b/c while a/b exists as a file.

Common situations: Deep destination trees where an earlier step wrote a file at what is now expected to be a directory level; path-layout changes between runs; generated destination paths colliding with old file outputs.

Related errors

• rename destination parent ${parent} is a file.
• Parent path is not a directory: " + parent
• Parent path is not a directory: {} {}
• rename destination parent {} is a file.
• Not a directory: {parent}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/4b519eab4e3f027a. Report an issue: GitHub.