apache/hadoop · critical · IOException

Failed to rename {} to {}

Error message

Failed to rename {} to {}

What it means

In the commit phase, FileOutputCommitter.mergePaths moves committed task output from the _temporary area to the final destination with fs.rename(from, to); a false return (failure without exception) becomes IOException('Failed to rename <from> to <to>') at FileOutputCommitter.java:483. Rename success is filesystem-dependent — it fails on permission errors, missing parents, destination-exists semantics, or object stores with non-atomic rename.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter.java:482

        false,
        "Merging data from %s to %s", from, to)) {
      reportProgress(context);
      FileStatus toStat;
      try {
        toStat = fs.getFileStatus(to);
      } catch (FileNotFoundException fnfe) {
        toStat = null;
      }

      if (from.isFile()) {
        if (toStat != null) {
          if (!fs.delete(to, true)) {
            throw new IOException("Failed to delete " + to);
          }
        }

        if (!fs.rename(from.getPath(), to)) {
          throw new IOException("Failed to rename " + from + " to " + to);
        }
      } else if (from.isDirectory()) {
        if (toStat != null) {
          if (!toStat.isDirectory()) {
            if (!fs.delete(to, true)) {
              throw new IOException("Failed to delete " + to);
            }
            renameOrMerge(fs, from, to, context);
          } else {
            //It is a directory so merge everything in the directories
            for (FileStatus subFrom : fs.listStatus(from.getPath())) {
              Path subTo = new Path(to, subFrom.getPath().getName());
              mergePaths(fs, subFrom, subTo, context);
            }
          }
        } else {
          renameOrMerge(fs, from, to, context);
        }

View on GitHub (pinned to 2add963021)

Solutions

1. Verify and fix permissions/ownership on the output directory (hadoop fs -chown/-chmod) for the submitting user, including the _temporary subtree
2. Clear or version the output path per run (hadoop fs -rm -r /out, or /out-$RUNID) to avoid exists-collisions during rename
3. Eliminate concurrent writers: unique output dirs per job/attempt, disable duplicate submissions, and let only one committer own a path
4. For object stores, switch to the store-native committer (fs.s3a.committer.name=magic|directory) or a zero-rename committer rather than FileOutputCommitter renames
5. Check cluster health at the commit timestamp in logs (safe mode, NN failover) and simply retry the job once healthy — commit is idempotent per attempt after cleanup

Example fix

# before
hadoop fs -chmod 555 /out            # read-only output dir
hadoop jar app.jar Driver /in /out   # task commit rename -> 'Failed to rename ... to ...'

# after
hadoop fs -chmod -R 755 /out
hadoop fs -rm -r /out/_temporary
hadoop jar app.jar Driver /in /out

# for S3, prefer a real committer:
# -Dfs.s3a.committer.name=magic -Dmapreduce.outputcommitter.factory.scheme.s3a=org.apache.hadoop.fs.s3a.commit.S3ACommitterFactory

static void requireCommittableOutput(Path out, Configuration conf) throws IOException {
  FileSystem fs = out.getFileSystem(conf);
  Path p = out.getParent() != null ? out.getParent() : out;
  if (fs.exists(p) && !fs.getFileStatus(p).getPermission().getUserAction().implies(FsAction.WRITE))
    throw new IllegalStateException("No write permission for output parent: " + p);
  if (fs.exists(new Path(out, "_temporary")))
    throw new IllegalStateException("Stale _temporary dir from a previous attempt; remove it before rerun");
}

catch (IOException e) { if (String.valueOf(e.getMessage()).startsWith("Failed to rename")) { // commit-phase rename failed: verify perms/safe-mode, remove stale _temporary, resubmit once
  throw new IOException("Commit rename failed (" + e.getMessage() + ") — check output perms, concurrent writers, FS health; clean and retry", e); } throw e; }

When it happens

Trigger: Committing a job whose output parent dir lacks write permission; the destination already exists as a file where a directory is needed (or vice versa, in the not-file branches of mergePaths); concurrent committers (AM + task, two jobs, speculative attempts) racing on the same target names; HDFS in safe mode or during NN failover; S3A/ABFS paths without proper committer configuration emulating rename via copy+delete.

Common situations: Output directory owned by another user or read-only (chmod 555); reruns into existing output with different file layout; classic MapReduce on object stores (S3) using the default committer; slow metadata operations timing out; algorithm.version=1 tasks committing while another attempt commits the same files.

Related errors

• Failed to delete {}
• Unable to recover task %s, output: %s
• Unable to rename {src} to {dst}: couldn't create parent dire
• Unable to rename {src} to {dst}
• Invalid state of the job for cleanup. State found " + jobRun

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/e5b2b5929f59b940. Report an issue: GitHub.