apache/beam · error · FileAlreadyExistsException

Unable to rename resource

Error message

Unable to rename resource %s to %s as destination already exists and couldn't be deleted.

What it means

If fs.rename() returns false and the destination still exists after Beam attempted fs.delete(destPath, false), rename() throws FileAlreadyExistsException with this message. The Hadoop rename refuses to overwrite an existing destination and Beam's non-recursive delete failed to clear it.

Solutions

Delete or move the existing destination before renaming.
Ensure the caller has delete permission on the destination path.
For directories, delete recursively or choose a fresh destination path.

Example fix

// before
fileSystem.rename(ImmutableList.of(src), ImmutableList.of(dest));
// after
try {
  fileSystem.delete(ImmutableList.of(dest));
} catch (FileNotFoundException ignored) {
}
fileSystem.rename(ImmutableList.of(src), ImmutableList.of(dest));

Defensive patterns

Strategy: validation

Validate before calling

if (fileSystem.match(dest.toString()).status() == MatchResult.Status.OK) {
  fileSystem.delete(ImmutableList.of(dest));
}

Try / catch

try {
  fileSystem.rename(srcs, dests);
} catch (FileAlreadyExistsException e) {
  fileSystem.delete(ImmutableList.of(dest));
  fileSystem.rename(srcs, dests);
}

Prevention

Clean or version output destinations before renames
Ensure the job user has delete permission on the destination
Use unique, timestamped destination paths on reruns

When it happens

Trigger: Calling HadoopFileSystem.rename() where the destination path already exists and the pre-emptive non-recursive delete of the destination fails (permissions, dest is a non-empty directory, etc.).

Common situations: Re-running a pipeline whose previous run left output files at the destination; renaming a file onto an existing directory; delete blocked by permissions or a non-empty dir.

Understand the failure class

Background: "already exists" / EEXIST / FileAlreadyExistsException: what the 'file already exists' error means and how to fix it — this error's family across 37 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/26bdc7d7962ffba9. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/io/hadoop-file-system/src/main/java/org/apache/beam/sdk/io/hdfs/HadoopFileSystem.java:285

      boolean success = fs.rename(srcPath, destPath);

      // If the failure was due to the file already existing, delete and retry (BEAM-5036).
      // This should be the exceptional case, so handle here rather than incur the overhead of
      // testing first
      if (!success && fs.exists(srcPath) && fs.exists(destPath)) {
        LOG.debug(LOG_DELETING_EXISTING_FILE, Path.getPathWithoutSchemeAndAuthority(destPath));
        fs.delete(destPath, false); // not recursive
        success = fs.rename(srcPath, destPath);
      }

      if (!success) {
        if (!fs.exists(srcPath)) {
          throw new FileNotFoundException(
              String.format(
                  "Unable to rename resource %s to %s as source not found.", srcPath, destPath));

        } else if (fs.exists(destPath)) {
          throw new FileAlreadyExistsException(
              String.format(
                  "Unable to rename resource %s to %s as destination already exists and couldn't be deleted.",
                  srcPath, destPath));

        } else {
          throw new IOException(
              String.format(
                  "Unable to rename resource %s to %s. No further information provided by underlying filesystem.",
                  srcPath, destPath));
        }
      }
    }
  }

  /** Ensures that the target directory exists for the given filePath. */
  private void mkdirs(Path filePath) throws IOException {
    final org.apache.hadoop.fs.FileSystem fs = filePath.getFileSystem(configuration);
    final Path targetDirectory = filePath.getParent();

View on GitHub (pinned to 12126d8942)