{"record":{"id":"a0445c05c8182ec9","repo":"apache/hadoop","slug":"cannot-overwrite-an-existing-file-s","errorCode":null,"errorMessage":"Cannot overwrite an existing file: %s","messagePattern":"Cannot overwrite an existing file: (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-cloud-storage-project/hadoop-gcp/src/main/java/org/apache/hadoop/fs/gs/GoogleCloudStorageFileSystem.java","lineNumber":560,"sourceCode":"      result.add(getFileInfo(path));\n    }\n\n    return result;\n  }\n\n  private URI getDstUri(FileInfo srcInfo, FileInfo dstInfo, @Nullable FileInfo dstParentInfo)\n      throws IOException {\n    URI src = srcInfo.getPath();\n    URI dst = dstInfo.getPath();\n\n    // Throw if src is a file and dst == GCS_ROOT\n    if (!srcInfo.isDirectory() && dst.equals(GCSROOT)) {\n      throw new IOException(\"A file cannot be created in root.\");\n    }\n\n    // Throw if the destination is a file that already exists, and it's not a source file.\n    if (dstInfo.exists() && !dstInfo.isDirectory() && (srcInfo.isDirectory() || !dst.equals(src))) {\n      throw new IOException(\"Cannot overwrite an existing file: \" + dst);\n    }\n\n    // Rename operation cannot be completed if parent of destination does not exist.\n    if (dstParentInfo != null && !dstParentInfo.exists()) {\n      throw new IOException(\n          \"Cannot rename because path does not exist: \" + dstParentInfo.getPath());\n    }\n\n    // Leaf item of the source path.\n    String srcItemName = getItemName(src);\n\n    // Having taken care of the initial checks, apply the regular rules.\n    // After applying the rules, we will be left with 2 paths such that:\n    // -- either both are files or both are directories\n    // -- src exists and dst leaf does not exist\n    if (srcInfo.isDirectory()) {\n      // -- if src is a directory\n      //    -- dst is an existing file => disallowed","sourceCodeStart":542,"sourceCodeEnd":578,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-cloud-storage-project/hadoop-gcp/src/main/java/org/apache/hadoop/fs/gs/GoogleCloudStorageFileSystem.java#L542-L578","documentation":"Thrown by GoogleCloudStorageFileSystem.getDstUri (hadoop-gcp) while computing the effective destination of a rename: the destination exists as a file AND either the source is a directory or the destination differs from the source. The GCS connector implements Hadoop rename semantics, which never overwrite an existing destination file, so it refuses the operation up front.","triggerScenarios":"Calling gcsFs.rename(srcUri, dstUri) or GoogleHadoopFileSystem.rename(Path, Path) where getFileInfo(dst) reports an existing file and (srcInfo.isDirectory() || !dst.equals(src)). Examples: renaming gs://b/a.txt onto existing file gs://b/b.txt; renaming a directory onto an existing file path; 'mv foo bar' style calls where bar is already a file.","commonSituations":"Pipelines that write to temp names then rename onto final names that already exist from a previous failed run; idempotent job retries that repeat a rename step; developers assuming POSIX 'mv' overwrite behavior; Hive/Spark commit phases moving staging files onto existing output paths.","solutions":["Delete or move the existing destination file first (fs.delete(dst, false) after fs.exists(dst) && !fs.getFileStatus(dst).isDirectory()).","Rename to a fresh, non-existent destination (append a unique suffix/UUID if the name matters).","If re-running a failed pipeline, clean up partial outputs before the rename stage.","Treat this IOException as a 'destination exists' signal in caller logic instead of retrying the identical rename."],"exampleFix":"// before\nPath src = new Path(\"gs://bucket/a.txt\");\nPath dst = new Path(\"gs://bucket/b.txt\"); // b.txt already exists -> IOException\nfs.rename(src, dst);\n\n// after\nif (fs.exists(dst) && !fs.getFileStatus(dst).isDirectory()) {\n  fs.delete(dst, false);\n}\nfs.rename(src, dst);","handlingStrategy":"validation","validationCode":"Path src = new Path(\"gs://bucket/a.txt\");\nPath dst = new Path(\"gs://bucket/b.txt\");\nFileSystem fs = src.getFileSystem(conf);\nif (fs.exists(dst) && !fs.getFileStatus(dst).isDirectory()) {\n  fs.delete(dst, false); // or pick a unique destination name instead\n}\nboolean renamed = fs.rename(src, dst);","typeGuard":null,"tryCatchPattern":"catch IOException around rename; check message startsWith(\"Cannot overwrite an existing file\") to distinguish a destination conflict from transient GCS errors - retrying the same rename will not help, so delete or rename aside the destination instead.","preventionTips":["Never assume POSIX mv overwrite semantics - Hadoop rename never overwrites an existing file on GCS.","Generate destination names deterministically-unique (or check exists) in retryable pipelines.","Clean up outputs of failed runs before re-running rename stages."],"tags":["rename","gcs","hadoop","filesystem","google-cloud-storage"],"backgroundTag":"rename-destination-exists","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}