apache/hadoop · error · IOException

Destination path %s already exist, cannot rename!

Error message

Destination path %s already exist, cannot rename!

What it means

Thrown by SFTPFileSystem.rename when the destination path already exists on the SFTP server (exists(channel, absoluteDst) is true). The adapter refuses overwrite-by-rename: unlike POSIX rename(2) and unlike HDFS (which returns false), SFTPFileSystem raises an IOException with the destination path in the message.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/SFTPFileSystem.java:479

   * @return rename successful?
   * @throws IOException
   */
  private boolean rename(ChannelSftp channel, Path src, Path dst)
      throws IOException {
    Path workDir;
    try {
      workDir = new Path(channel.pwd());
    } catch (SftpException e) {
      throw new IOException(e);
    }
    Path absoluteSrc = makeAbsolute(workDir, src);
    Path absoluteDst = makeAbsolute(workDir, dst);

    if (!exists(channel, absoluteSrc)) {
      throw new IOException(String.format(E_SPATH_NOTEXIST, src));
    }
    if (exists(channel, absoluteDst)) {
      throw new IOException(String.format(E_DPATH_EXIST, dst));
    }
    boolean renamed = true;
    try {
      final String previousCwd = channel.pwd();
      channel.cd("/");
      channel.rename(src.toUri().getPath(), dst.toUri().getPath());
      channel.cd(previousCwd);
    } catch (SftpException e) {
      renamed = false;
    }
    return renamed;
  }

  @Override
  public void initialize(URI uriInfo, Configuration conf) throws IOException {
    super.initialize(uriInfo, conf);

    setConfigurationFromURI(uriInfo, conf);

View on GitHub (pinned to 2add963021)

Solutions

1. Delete the destination before renaming: if (fs.exists(dst)) { fs.delete(dst, false); } (use recursive=true only if replacing a directory is intended).
2. Generate unique destination names (append a timestamp or UUID) when the rename may run more than once.
3. For idempotent pipelines, catch the IOException and treat 'Destination path ... already exist' as 'move already completed' rather than a failure.

Example fix

// before
fs.rename(src, dst); // throws if dst exists

// after
if (fs.exists(dst)) {
  fs.delete(dst, false);
}
fs.rename(src, dst);

if (fs.exists(dst)) {
  fs.delete(dst, false); // deliberate clobber; use true only to replace a directory
}
boolean ok = fs.rename(src, dst);

try {
  fs.rename(src, dst);
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().contains("already exist")) {
    // destination present: idempotent no-op
  } else {
    throw e;
  }
}

When it happens

Trigger: fs.rename(src, dst) where dst names an existing file or an existing directory on the SFTP server; also retry loops that re-run a 'move to archive' step after a previous attempt already created the destination.

Common situations: Re-running an idempotent ingestion job that moves files into an archive directory that already holds a file with the same name; renaming onto an existing directory; porting scripts from a filesystem where rename silently replaces the destination.

Related errors

• Source path %s does not exist
• File: %s already exists
• Cannot overwrite an existing file: %s
• Cannot rename because path does not exist: %s
• new destination is an existed directory

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/16cf1833a9b28687. Report an issue: GitHub.