apache/hadoop · error · IOException

Could not rename {oldDir} to {newDir}

Error message

Could not rename {oldDir} to {newDir}

What it means

Thrown by MapFile.rename(FileSystem, String, String) when FileSystem.rename() returns false. A MapFile is a directory containing data and index files, so 'renaming the map' means renaming the whole directory; the underlying FS refused. On HDFS the classic refusal reason is that the destination already exists — its rename is not POSIX-overwrite.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/MapFile.java:903

      }
      data.close();
    }

  }

  /**
   * Renames an existing map directory.
   * @param fs fs.
   * @param oldName oldName.
   * @param newName newName.
   * @throws IOException raised on errors performing I/O.
   */
  public static void rename(FileSystem fs, String oldName, String newName)
    throws IOException {
    Path oldDir = new Path(oldName);
    Path newDir = new Path(newName);
    if (!fs.rename(oldDir, newDir)) {
      throw new IOException("Could not rename " + oldDir + " to " + newDir);
    }
  }

  /**
   * Deletes the named map file.
   * @param fs input fs.
   * @param name input name.
   * @throws IOException raised on errors performing I/O.
   */
  public static void delete(FileSystem fs, String name) throws IOException {
    Path dir = new Path(name);
    Path data = new Path(dir, DATA_FILE_NAME);
    Path index = new Path(dir, INDEX_FILE_NAME);

    fs.delete(data, true);
    fs.delete(index, true);
    fs.delete(dir, true);
  }

View on GitHub (pinned to 2add963021)

Solutions

1. Delete or move the existing destination first: if (fs.exists(newDir)) fs.delete(newDir, true) — only if overwrite is intended.
2. Verify the source exists and you have write permission on both source and destination parent (hadoop fs -ls, -chmod).
3. Ensure the destination parent directory exists before renaming.
4. Wrap in try-catch (IOException) and surface both paths (already in the message) for the failure report.

Example fix

// before: second run throws because target exists
MapFile.rename(fs, "/out/part-0", "/out/final");

// after: HDFS rename does not overwrite — clear the target explicitly
Path dst = new Path("/out/final");
if (fs.exists(dst)) {
  fs.delete(dst, true);
}
MapFile.rename(fs, "/out/part-0", dst.toString());

Path dst = new Path(newName);
if (fs.exists(dst)) {
  if (!overwrite) throw new IOException(dst + " already exists");
  fs.delete(dst, true); // HDFS rename will not overwrite
}
MapFile.rename(fs, oldName, newName);

try {
  MapFile.rename(fs, oldName, newName);
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().startsWith("Could not rename")) {
    // destination exists, source missing, or perms — diagnose both paths
    if (fs.exists(new Path(newName))) fs.delete(new Path(newName), true);
    MapFile.rename(fs, oldName, newName); // retry once after clearing target
  } else { throw e; }
}

When it happens

Trigger: Calling MapFile.rename where the destination directory already exists (HDFS rename does not overwrite); source path missing; user lacks permission on source or parent of destination; destination parent doesn't exist.

Common situations: Committing map output by renaming part-r dir onto an existing final dir from a previous run (re-run without cleaning output); atomic-commit patterns ported from local FS assumptions to HDFS; permission changes between runs.

Related errors

• Mkdirs failed to create directory {dirName}
• {} doesn't support renameSnapshot
• Cannot Rename within internal dirs of mount table: src={} is
• Cannot Rename within internal dirs of mount table: dest={} i
• key class or comparator option must be set

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/5f68a1b068102a0b. Report an issue: GitHub.