apache/hadoop · error · FileAlreadyExistsException

Failed to rename %s to %s, file already exists or not empty!

Error message

Failed to rename %s to %s, file already exists or not empty!

What it means

Thrown by AliyunOSSFileSystem.rename(): the destination exists and is a directory, so the source is renamed into it as dst/srcName — but listStatus on that joined path found existing entries. To honor POSIX rename-on-directory semantics (destination directory must be empty), it throws FileAlreadyExistsException("Failed to rename %s to %s, file already exists or not empty!").

Source

Thrown at hadoop-tools/hadoop-aliyun/src/main/java/org/apache/hadoop/fs/aliyun/oss/AliyunOSSFileSystem.java:665

        throw new IOException(String.format(
            "Failed to rename %s to %s, %s is a file", srcPath, dstPath,
            dstPath.getParent()));
      }
    } else {
      if (srcStatus.getPath().equals(dstStatus.getPath())) {
        return !srcStatus.isDirectory();
      } else if (dstStatus.isDirectory()) {
        // If dst is a directory
        dstPath = new Path(dstPath, srcPath.getName());
        FileStatus[] statuses;
        try {
          statuses = listStatus(dstPath);
        } catch (FileNotFoundException fnde) {
          statuses = null;
        }
        if (statuses != null && statuses.length > 0) {
          // If dst exists and not a directory / not empty
          throw new FileAlreadyExistsException(String.format(
              "Failed to rename %s to %s, file already exists or not empty!",
              srcPath, dstPath));
        }
      } else {
        // If dst is not a directory
        throw new FileAlreadyExistsException(String.format(
            "Failed to rename %s to %s, file already exists!", srcPath,
            dstPath));
      }
    }

    boolean succeed;
    if (srcStatus.isDirectory()) {
      succeed = copyDirectory(srcPath, dstPath);
    } else {
      succeed = copyFile(srcPath, srcStatus.getLen(), dstPath);
    }

View on GitHub (pinned to 2add963021)

Solutions

Delete the existing entry at dst/srcName before renaming, or choose a fresh destination name
Make rename targets idempotent-unique (attempt id / timestamp suffix) so reruns cannot collide
If the leftover came from a failed commit, clean the destination subtree before the retry

Example fix

// before
fs.rename(new Path("oss://b/src"), new Path("oss://b/dst")); // dst/src exists

// after
Path target = new Path("oss://b/dst/src");
if (fs.exists(target)) {
  fs.delete(target, false);
}
fs.rename(new Path("oss://b/src"), new Path("oss://b/dst"));

Defensive patterns

Strategy: validation

Validate before calling

Path target = dst;
if (fs.exists(dst) && fs.getFileStatus(dst).isDirectory()) {
  target = new Path(dst, src.getName());
}
if (fs.exists(target)) {
  throw new IOException("Rename target already populated: " + target);
}
fs.rename(src, dst);

Try / catch

catch (FileAlreadyExistsException e) { /* dst/srcName exists: delete it or pick a fresh name, then retry once */ throw e; }

Prevention

Pre-compute the effective destination (dst/srcName when dst is a dir) and check emptiness before rename
Make commit/move stages idempotent with unique target names
Clean partially-committed destinations before reruns

When it happens

Trigger: Calling rename(src, dstDir) where an object already exists at dstDir/srcName; e.g., rename(oss://b/src, oss://b/dst) when object 'dst/src' already exists — common in Hive partition moves and committer staging where the target was partially populated.

Common situations: Rerunning a job after a partially failed commit that left the target in place; concurrent writers moving files into the same directory; incremental pipelines that assume overwrite-on-rename (OSS rename is copy+delete and does not overwrite).

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/c6563d084c218e16. Report an issue: GitHub.