apache/hadoop · error · RenameFailedException

destination is an existed file

Error message

destination is an existed file

What it means

When the destination of a rename exists and is a file (not a directory), Hadoop semantics forbid silently overwriting it: if the source and destination keys differ, renameBasedOnObject throws RenameFailedException('destination is an existed file').withExitCode(false). Only the exact src==dst case is tolerated with a warning and a true return.

Source

Thrown at hadoop-cloud-storage-project/hadoop-huaweicloud/src/main/java/org/apache/hadoop/fs/obs/OBSObjectBucketUtils.java:144

        if (dstStatus.isDirectory()) {
          throw new RenameFailedException(src, dst,
              "new destination is an existed directory")
              .withExitCode(false);
        } else {
          throw new RenameFailedException(src, dst,
              "new destination is an existed file")
              .withExitCode(false);
        }
      } else {

        if (srcKey.equals(dstKey)) {
          LOG.warn(
              "rename: src and dest refer to the same file or"
                  + " directory: {}",
              dst);
          return true;
        } else {
          throw new RenameFailedException(src, dst,
              "destination is an existed file")
              .withExitCode(false);
        }
      }
    } catch (FileNotFoundException e) {
      LOG.debug("rename: destination path {} not found", dst);

      // Parent must exist
      checkDestinationParent(owner, src, dst);
    }

    if (dstKey.startsWith(srcKey)
        && dstKey.charAt(srcKey.length()) == Path.SEPARATOR_CHAR) {
      LOG.error("rename: dest [{}] cannot be a descendant of src [{}]",
          dst, src);
      return false;
    }

View on GitHub (pinned to 2add963021)

Solutions

1. Delete the existing destination file first when overwrite is intended: fs.delete(dst, false)
2. Make the commit idempotent: if the destination exists and the source is gone, treat the commit as done
3. Give each run a unique output path and let a downstream step pick the latest

Example fix

// before
fs.rename(tmpPath, finalPath); // fails: destination is an existed file

// after
if (fs.exists(finalPath)) {
  fs.delete(finalPath, false);
}
fs.rename(tmpPath, finalPath);

if (fs.exists(dst)) {
  if (fs.getFileStatus(dst).isFile()) {
    fs.delete(dst, false); // overwrite intended
  } else {
    throw new IOException("Destination exists and is not a file: " + dst);
  }
}

try {
  fs.rename(tmpPath, finalPath);
} catch (RenameFailedException e) {
  if (e.getMessage().contains("destination is an existed file")) {
    // prior run committed already: delete or skip, then retry
  } else {
    throw e;
  }
}

When it happens

Trigger: fs.rename('/a/tmp-file', '/b/final-file') where /b/final-file is an existing different file; atomic commit patterns (write temp, rename to final) re-run when the final file already exists; two writers committing to the same final path.

Common situations: Job retry after a crash that happened after commit but before acknowledgment; output committers that assume rename-over-file works as on a local filesystem; multiple pipelines writing the same output key.

Related errors

• new destination is an existed file
• new destination is an existed directory
• File: %s already exists
• {} is root directory
• destination parent [{}] is not a directory

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/51f8051a5a5541b6. Report an issue: GitHub.