apache/hadoop · error · FileAlreadyExistsException

rename destination ${dst} already exists.

Error message

rename destination ${dst} already exists.

What it means

In the deprecated options-aware rename: the destination exists and Rename.OVERWRITE was not among the options, so the default implementation refuses with FileAlreadyExistsException('rename destination dst already exists.') - POSIX-style protection against silently clobbering an existing target.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java:1699

        if (option == Rename.OVERWRITE) {
          overwrite = true;
        }
      }
    }

    FileStatus dstStatus;
    try {
      dstStatus = getFileLinkStatus(dst);
    } catch (IOException e) {
      dstStatus = null;
    }
    if (dstStatus != null) {
      if (srcStatus.isDirectory() != dstStatus.isDirectory()) {
        throw new IOException("Source " + src + " Destination " + dst
            + " both should be either file or directory");
      }
      if (!overwrite) {
        throw new FileAlreadyExistsException("rename destination " + dst
            + " already exists.");
      }
      // Delete the destination that is a file or an empty directory
      if (dstStatus.isDirectory()) {
        FileStatus[] list = listStatus(dst);
        if (list != null && list.length != 0) {
          throw new IOException(
              "rename cannot overwrite non empty destination directory " + dst);
        }
      }
      delete(dst, false);
    } else {
      final Path parent = dst.getParent();
      final FileStatus parentStatus = getFileStatus(parent);
      if (parentStatus == null) {
        throw new FileNotFoundException("rename destination parent " + parent
            + " not found.");
      }

View on GitHub (pinned to 2add963021)

Solutions

1. If overwriting is intended, pass Rename.OVERWRITE
2. Treat FileAlreadyExistsException as 'already published' and skip (idempotent commit)
3. Otherwise move or delete the existing destination first

Example fix

// before
fs.rename(src, dst); // dst exists -> FileAlreadyExistsException

// after
try {
  fs.rename(src, dst, Rename.OVERWRITE);
} catch (FileAlreadyExistsException e) {
  LOG.info("already published: {}", dst); // idempotent rerun
}

if (fs.exists(dst)) {
  if (!overwrite) {
    LOG.info("destination already exists, treating as done: {}", dst);
    return;
  }
}
fs.rename(src, dst, overwrite ? new Rename[]{Rename.OVERWRITE} : new Rename[0]);

When it happens

Trigger: rename(src, dst, ...) without Rename.OVERWRITE while dst exists; publish/commit flows re-running against a previous run's output.

Common situations: Idempotent pipeline reruns; atomic-publish patterns (rename temp -> final) where the final path already exists from an earlier successful run.

Related errors

• Cannot overwrite an existing file: %s
• Cannot rename because path does not exist: %s
• Rename unsuccessful : '%s' to '%s'
• The source {src} and destination {dst} are the same
• File already exists: ${path}. Append or overwrite option mus

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/34d87c3c12be665d. Report an issue: GitHub.