apache/hadoop · error · IOException

Cannot overwrite an existing file: %s

Error message

Cannot overwrite an existing file: %s

What it means

Thrown by GoogleCloudStorageFileSystem.getDstUri (hadoop-gcp) while computing the effective destination of a rename: the destination exists as a file AND either the source is a directory or the destination differs from the source. The GCS connector implements Hadoop rename semantics, which never overwrite an existing destination file, so it refuses the operation up front.

Source

Thrown at hadoop-cloud-storage-project/hadoop-gcp/src/main/java/org/apache/hadoop/fs/gs/GoogleCloudStorageFileSystem.java:560

      result.add(getFileInfo(path));
    }

    return result;
  }

  private URI getDstUri(FileInfo srcInfo, FileInfo dstInfo, @Nullable FileInfo dstParentInfo)
      throws IOException {
    URI src = srcInfo.getPath();
    URI dst = dstInfo.getPath();

    // Throw if src is a file and dst == GCS_ROOT
    if (!srcInfo.isDirectory() && dst.equals(GCSROOT)) {
      throw new IOException("A file cannot be created in root.");
    }

    // Throw if the destination is a file that already exists, and it's not a source file.
    if (dstInfo.exists() && !dstInfo.isDirectory() && (srcInfo.isDirectory() || !dst.equals(src))) {
      throw new IOException("Cannot overwrite an existing file: " + dst);
    }

    // Rename operation cannot be completed if parent of destination does not exist.
    if (dstParentInfo != null && !dstParentInfo.exists()) {
      throw new IOException(
          "Cannot rename because path does not exist: " + dstParentInfo.getPath());
    }

    // Leaf item of the source path.
    String srcItemName = getItemName(src);

    // Having taken care of the initial checks, apply the regular rules.
    // After applying the rules, we will be left with 2 paths such that:
    // -- either both are files or both are directories
    // -- src exists and dst leaf does not exist
    if (srcInfo.isDirectory()) {
      // -- if src is a directory
      //    -- dst is an existing file => disallowed

View on GitHub (pinned to 2add963021)

Solutions

  1. Delete or move the existing destination file first (fs.delete(dst, false) after fs.exists(dst) && !fs.getFileStatus(dst).isDirectory()).
  2. Rename to a fresh, non-existent destination (append a unique suffix/UUID if the name matters).
  3. If re-running a failed pipeline, clean up partial outputs before the rename stage.
  4. Treat this IOException as a 'destination exists' signal in caller logic instead of retrying the identical rename.

Example fix

// before
Path src = new Path("gs://bucket/a.txt");
Path dst = new Path("gs://bucket/b.txt"); // b.txt already exists -> IOException
fs.rename(src, dst);

// after
if (fs.exists(dst) && !fs.getFileStatus(dst).isDirectory()) {
  fs.delete(dst, false);
}
fs.rename(src, dst);
Defensive patterns

Strategy: validation

Validate before calling

Path src = new Path("gs://bucket/a.txt");
Path dst = new Path("gs://bucket/b.txt");
FileSystem fs = src.getFileSystem(conf);
if (fs.exists(dst) && !fs.getFileStatus(dst).isDirectory()) {
  fs.delete(dst, false); // or pick a unique destination name instead
}
boolean renamed = fs.rename(src, dst);

Try / catch

catch IOException around rename; check message startsWith("Cannot overwrite an existing file") to distinguish a destination conflict from transient GCS errors - retrying the same rename will not help, so delete or rename aside the destination instead.

Prevention

When it happens

Trigger: Calling gcsFs.rename(srcUri, dstUri) or GoogleHadoopFileSystem.rename(Path, Path) where getFileInfo(dst) reports an existing file and (srcInfo.isDirectory() || !dst.equals(src)). Examples: renaming gs://b/a.txt onto existing file gs://b/b.txt; renaming a directory onto an existing file path; 'mv foo bar' style calls where bar is already a file.

Common situations: Pipelines that write to temp names then rename onto final names that already exist from a previous failed run; idempotent job retries that repeat a rename step; developers assuming POSIX 'mv' overwrite behavior; Hive/Spark commit phases moving staging files onto existing output paths.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/a0445c05c8182ec9. Report an issue: GitHub.