What causes "Cannot rename because path does not exist: %s" in apache/hadoop?

Create the destination parent before renaming: fs.mkdirs(dst.getParent()) (GCS mkdirs creates placeholder objects).

apache/hadoop · error · IOException

Cannot rename because path does not exist: %s

Error message

Cannot rename because path does not exist: %s

What it means

getDstUri validates that the parent of the rename destination exists (dstParentInfo != null && !dstParentInfo.exists()). GCS has no real server-side directories, so the connector checks client-side that the destination's parent path is present (as an object, placeholder, or inferred directory); if not, the rename is rejected.

Source

Thrown at hadoop-cloud-storage-project/hadoop-gcp/src/main/java/org/apache/hadoop/fs/gs/GoogleCloudStorageFileSystem.java:565

  private URI getDstUri(FileInfo srcInfo, FileInfo dstInfo, @Nullable FileInfo dstParentInfo)
      throws IOException {
    URI src = srcInfo.getPath();
    URI dst = dstInfo.getPath();

    // Throw if src is a file and dst == GCS_ROOT
    if (!srcInfo.isDirectory() && dst.equals(GCSROOT)) {
      throw new IOException("A file cannot be created in root.");
    }

    // Throw if the destination is a file that already exists, and it's not a source file.
    if (dstInfo.exists() && !dstInfo.isDirectory() && (srcInfo.isDirectory() || !dst.equals(src))) {
      throw new IOException("Cannot overwrite an existing file: " + dst);
    }

    // Rename operation cannot be completed if parent of destination does not exist.
    if (dstParentInfo != null && !dstParentInfo.exists()) {
      throw new IOException(
          "Cannot rename because path does not exist: " + dstParentInfo.getPath());
    }

    // Leaf item of the source path.
    String srcItemName = getItemName(src);

    // Having taken care of the initial checks, apply the regular rules.
    // After applying the rules, we will be left with 2 paths such that:
    // -- either both are files or both are directories
    // -- src exists and dst leaf does not exist
    if (srcInfo.isDirectory()) {
      // -- if src is a directory
      //    -- dst is an existing file => disallowed
      //    -- dst is a directory => rename the directory.

      // The first case (dst is an existing file) is already checked earlier.
      // If the destination path looks like a file, make it look like a
      // directory path. This is because users often type 'mv foo bar'

View on GitHub (pinned to 2add963021)

Solutions

Create the destination parent before renaming: fs.mkdirs(dst.getParent()) (GCS mkdirs creates placeholder objects).
Rename to a destination directly under an existing directory instead of a new subtree.
Guard with fs.exists(dst.getParent()) and fail fast with a clear application error.

Example fix

// before
fs.rename(new Path("gs://bucket/src"), new Path("gs://bucket/newdir/dst"));
// -> IOException: Cannot rename because path does not exist: gs://bucket/newdir

// after
Path dst = new Path("gs://bucket/newdir/dst");
if (!fs.exists(dst.getParent())) {
  fs.mkdirs(dst.getParent());
}
fs.rename(new Path("gs://bucket/src"), dst);

Defensive patterns

Strategy: validation

Validate before calling

Path dst = new Path("gs://bucket/newdir/result");
if (!fs.exists(dst.getParent())) {
  fs.mkdirs(dst.getParent()); // creates placeholder objects for GCS directories
}
fs.rename(src, dst);

Try / catch

catch IOException with message contains("Cannot rename because path does not exist") - the message names the missing path; create it with mkdirs and retry once.

Prevention

Pre-create destination directory trees with mkdirs before rename stages - GCS directories are implicit.
Remember GCS has no server-side mkdir; the connector validates parents client-side.
In concurrent pipelines, re-verify parents just before renaming.

When it happens

Trigger: gcsFs.rename(src, dst) where UriPaths.getParentPath(dst) does not exist, e.g. rename gs://b/src to gs://b/newdir/dst when gs://b/newdir has never been created or written to.

Common situations: Destination subdirectories not pre-created because GCS directories are implicit (no placeholder until something is written under them); scripts ported from HDFS assuming rename creates intermediate directories; fresh buckets receiving their first structured data.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/66903901fae73a9f. Report an issue: GitHub.