apache/hadoop · error · RenameFailedException

new destination is an existed file

Error message

new destination is an existed file

What it means

When the destination of a rename exists and is a directory, the effective target becomes dst/<srcname>. renameBasedObject stats that composed key and, if it is an existing file, throws RenameFailedException('new destination is an existed file').withExitCode(false). Object storage cannot atomically overwrite a file during a directory-style rename, so the operation is refused.

Source

Thrown at hadoop-cloud-storage-project/hadoop-huaweicloud/src/main/java/org/apache/hadoop/fs/obs/OBSObjectBucketUtils.java:131

      // if there is no destination entry, an exception is raised.
      // hence this code sequence can assume that there is something
      // at the end of the path; the only detail being what it is and
      // whether or not it can be the destination of the rename.
      if (dstStatus.isDirectory()) {
        String newDstKey = OBSCommonUtils.maybeAddTrailingSlash(dstKey);
        String filename = srcKey.substring(
            OBSCommonUtils.pathToKey(owner, src.getParent()).length()
                + 1);
        newDstKey = newDstKey + filename;
        dstKey = newDstKey;
        dstStatus = owner.getFileStatus(
            OBSCommonUtils.keyToPath(dstKey));
        if (dstStatus.isDirectory()) {
          throw new RenameFailedException(src, dst,
              "new destination is an existed directory")
              .withExitCode(false);
        } else {
          throw new RenameFailedException(src, dst,
              "new destination is an existed file")
              .withExitCode(false);
        }
      } else {

        if (srcKey.equals(dstKey)) {
          LOG.warn(
              "rename: src and dest refer to the same file or"
                  + " directory: {}",
              dst);
          return true;
        } else {
          throw new RenameFailedException(src, dst,
              "destination is an existed file")
              .withExitCode(false);
        }
      }
    } catch (FileNotFoundException e) {

View on GitHub (pinned to 2add963021)

Solutions

1. Check for dst/<srcname> with getFileStatus before renaming and delete it when overwrite is intended
2. Treat 'target file exists and source still exists' as a signal of a prior partial run: verify content, then skip or delete
3. Write to unique per-attempt names and finalize with a metadata step instead of relying on overwrite

Example fix

// before
fs.rename(new Path("/staging/part-1"), new Path("/warehouse/db")); // /warehouse/db/part-1 already exists as a file

// after
Path target = new Path(new Path("/warehouse/db"), "part-1");
if (fs.exists(target)) {
  fs.delete(target, false);
}
fs.rename(new Path("/staging/part-1"), new Path("/warehouse/db"));

Path effectiveTarget = dst;
if (fs.exists(dst) && fs.getFileStatus(dst).isDirectory()) {
  effectiveTarget = new Path(dst, src.getName());
}
if (fs.exists(effectiveTarget) && fs.getFileStatus(effectiveTarget).isFile()) {
  fs.delete(effectiveTarget, false); // deliberate overwrite
}

try {
  fs.rename(src, dst);
} catch (RenameFailedException e) {
  if (e.getMessage().contains("new destination is an existed file")) {
    // delete dst/<srcname> if overwrite is intended, then retry once
  } else {
    throw e;
  }
}

When it happens

Trigger: fs.rename('/src/file', '/dst') when /dst/file already exists as a file; two clients moving different files named identically into the same directory; commit-by-rename re-executed after the destination file was already written.

Common situations: Write-once-then-commit pipelines on object storage where POSIX overwrite semantics were assumed; concurrent Spark task attempts committing to the same output subdirectory; retry after a network failure where the first attempt actually succeeded.

Related errors

• destination is an existed file
• new destination is an existed directory
• File: %s already exists
• {} is root directory
• destination parent [{}] is not a directory

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/d9184fe88c011a67. Report an issue: GitHub.