apache/hadoop · error · RenameFailedException

new destination is an existed directory

Error message

new destination is an existed directory

What it means

Under Hadoop rename semantics, when the destination exists and is a directory the source is moved inside it as dst/<srcname>. renameBasedOnObject builds that key, stats it, and when it resolves to an existing directory throws RenameFailedException('new destination is an existed directory').withExitCode(false). The false exit code marks the failure as permanent, not retryable.

Source

Thrown at hadoop-cloud-storage-project/hadoop-huaweicloud/src/main/java/org/apache/hadoop/fs/obs/OBSObjectBucketUtils.java:127

    FileStatus dstStatus;
    try {
      dstStatus = owner.getFileStatus(dst);
      // if there is no destination entry, an exception is raised.
      // hence this code sequence can assume that there is something
      // at the end of the path; the only detail being what it is and
      // whether or not it can be the destination of the rename.
      if (dstStatus.isDirectory()) {
        String newDstKey = OBSCommonUtils.maybeAddTrailingSlash(dstKey);
        String filename = srcKey.substring(
            OBSCommonUtils.pathToKey(owner, src.getParent()).length()
                + 1);
        newDstKey = newDstKey + filename;
        dstKey = newDstKey;
        dstStatus = owner.getFileStatus(
            OBSCommonUtils.keyToPath(dstKey));
        if (dstStatus.isDirectory()) {
          throw new RenameFailedException(src, dst,
              "new destination is an existed directory")
              .withExitCode(false);
        } else {
          throw new RenameFailedException(src, dst,
              "new destination is an existed file")
              .withExitCode(false);
        }
      } else {

        if (srcKey.equals(dstKey)) {
          LOG.warn(
              "rename: src and dest refer to the same file or"
                  + " directory: {}",
              dst);
          return true;
        } else {
          throw new RenameFailedException(src, dst,
              "destination is an existed file")

View on GitHub (pinned to 2add963021)

Solutions

1. Check whether dst/<srcname> exists before the rename and delete or merge it deliberately
2. Skip the rename when the target already holds the correct content, making the job idempotent
3. Use a unique destination name (timestamp or run id) when name collisions are possible

Example fix

// before
fs.rename(new Path("/staging/part-1"), new Path("/warehouse/db")); // /warehouse/db/part-1 already exists as a dir

// after
Path target = new Path(new Path("/warehouse/db"), "part-1");
if (fs.exists(target)) {
  fs.delete(target, true); // or skip if already ingested
}
fs.rename(new Path("/staging/part-1"), new Path("/warehouse/db"));

Path effectiveTarget = dst;
if (fs.exists(dst) && fs.getFileStatus(dst).isDirectory()) {
  effectiveTarget = new Path(dst, src.getName());
}
if (fs.exists(effectiveTarget) && fs.getFileStatus(effectiveTarget).isDirectory()) {
  throw new IOException("Target already exists as a directory: " + effectiveTarget);
}

try {
  fs.rename(src, dst);
} catch (RenameFailedException e) {
  if (e.getMessage().contains("new destination is an existed directory")) {
    // permanent failure: clean or merge dst/<srcname>, then decide
  } else {
    throw e;
  }
}

When it happens

Trigger: fs.rename('/src/d1', '/dst') when /dst/d1 already exists as a directory; concurrent jobs moving different sources into the same destination directory with colliding names; re-running a move after a first attempt already created the target directory.

Common situations: Date-partitioned layouts where a partition directory is moved into an existing tree; ingestion jobs that mkdirs the target first (which silently succeeds on OBS) and then rename into it; non-idempotent commit logic re-executed after a partial failure.

Related errors

• new destination is an existed file
• destination is an existed file
• File: %s already exists
• {} is root directory
• destination parent [{}] is not a directory

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/48b983a5c725743f. Report an issue: GitHub.