apache/hadoop · error · FileAlreadyExistsException

Failed to rename %s to %s, file already exists!

Error message

Failed to rename %s to %s, file already exists!

What it means

Thrown by AliyunOSSFileSystem.rename(): the destination path exists and is a file (not a directory). Hadoop rename semantics do not overwrite an existing file in this connector, so it fails with FileAlreadyExistsException("Failed to rename %s to %s, file already exists!") and the source is left untouched.

Source

Thrown at hadoop-tools/hadoop-aliyun/src/main/java/org/apache/hadoop/fs/aliyun/oss/AliyunOSSFileSystem.java:671

        return !srcStatus.isDirectory();
      } else if (dstStatus.isDirectory()) {
        // If dst is a directory
        dstPath = new Path(dstPath, srcPath.getName());
        FileStatus[] statuses;
        try {
          statuses = listStatus(dstPath);
        } catch (FileNotFoundException fnde) {
          statuses = null;
        }
        if (statuses != null && statuses.length > 0) {
          // If dst exists and not a directory / not empty
          throw new FileAlreadyExistsException(String.format(
              "Failed to rename %s to %s, file already exists or not empty!",
              srcPath, dstPath));
        }
      } else {
        // If dst is not a directory
        throw new FileAlreadyExistsException(String.format(
            "Failed to rename %s to %s, file already exists!", srcPath,
            dstPath));
      }
    }

    boolean succeed;
    if (srcStatus.isDirectory()) {
      succeed = copyDirectory(srcPath, dstPath);
    } else {
      succeed = copyFile(srcPath, srcStatus.getLen(), dstPath);
    }

    return srcPath.equals(dstPath) || (succeed && delete(srcPath, true));
  }

  /**
   * Copy file from source path to destination path.
   * (the caller should make sure srcPath is a file and dstPath is valid)

View on GitHub (pinned to 2add963021)

Solutions

  1. Delete the existing destination object first if replacement is intended, then rename
  2. Design moves to be idempotent: check fs.exists(dst) and skip/rename-with-suffix instead of colliding
  3. Clean stale outputs from prior runs before the move stage

Example fix

// before
fs.rename(src, dst); // dst exists as a file

// after
if (fs.exists(dst)) {
  fs.delete(dst, false);
}
fs.rename(src, dst);
Defensive patterns

Strategy: validation

Validate before calling

if (fs.exists(dst) && fs.getFileStatus(dst).isFile()) {
  boolean replace = /* policy */ false;
  if (!replace) throw new IOException("Destination exists: " + dst);
  fs.delete(dst, false);
}
fs.rename(src, dst);

Try / catch

catch (FileAlreadyExistsException e) { /* destination file exists: apply replace-or-skip policy explicitly */ throw e; }

Prevention

When it happens

Trigger: Calling rename(src, dstFile) when an object already exists at dstFile; e.g., re-running a move phase that first succeeded, or archive layouts where source and destination names coincide on retry.

Common situations: Non-idempotent pipelines rerun after partial completion; commit retry moving an already-committed file; users expecting 'mv -f' overwrite behavior, which OSS rename does not implement.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/e3ec243829b6cabe. Report an issue: GitHub.