apache/hadoop · error · IOException

Could not rename {} to {}

Error message

Could not rename {} to {}

What it means

Thrown from FileOutputCommitter.commitTask (FileOutputCommitter.java:606) under commit algorithm v1 after the (optional) delete of an old committed-task dir succeeded but the subsequent fs.rename(taskAttemptPath, committedTaskPath) returned false. This is the actual move of task output from <outdir>/_temporary/<appAttemptId>/attempt_*/ into the committed-task slot; a false return means the FS rejected the rename.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter.java:606

      }
      FileSystem fs = taskAttemptPath.getFileSystem(context.getConfiguration());
      FileStatus taskAttemptDirStatus;
      try {
        taskAttemptDirStatus = fs.getFileStatus(taskAttemptPath);
      } catch (FileNotFoundException e) {
        taskAttemptDirStatus = null;
      }

      if (taskAttemptDirStatus != null) {
        if (algorithmVersion == 1) {
          Path committedTaskPath = getCommittedTaskPath(context);
          if (fs.exists(committedTaskPath)) {
             if (!fs.delete(committedTaskPath, true)) {
               throw new IOException("Could not delete " + committedTaskPath);
             }
          }
          if (!fs.rename(taskAttemptPath, committedTaskPath)) {
            throw new IOException("Could not rename " + taskAttemptPath + " to "
                + committedTaskPath);
          }
          LOG.info("Saved output of task '" + attemptId + "' to " +
              committedTaskPath);
        } else {
          // directly merge everything from taskAttemptPath to output directory
          mergePaths(fs, taskAttemptDirStatus, outputPath, context);
          LOG.info("Saved output of task '" + attemptId + "' to " +
              outputPath);

          if (context.getConfiguration().getBoolean(
              FILEOUTPUTCOMMITTER_TASK_CLEANUP_ENABLED,
              FILEOUTPUTCOMMITTER_TASK_CLEANUP_ENABLED_DEFAULT)) {
            LOG.debug(String.format(
                "Deleting the temporary directory of '%s': '%s'",
                attemptId, taskAttemptPath));
            if(!fs.delete(taskAttemptPath, true)) {
              LOG.warn("Could not delete " + taskAttemptPath);

View on GitHub (pinned to 2add963021)

Solutions

1. Confirm <outdir>/_temporary/<appAttemptId> exists at commit time; if it was deleted mid-job, rerun into a fresh output directory
2. Verify write permission on the _temporary tree for the job user and fix ownership (hadoop fs -chown -R)
3. Check HDFS quotas/disk (hadoop fs -count -q <outdir>) and free space
4. Move to mapreduce.fileoutputcommitter.algorithm.version=2, which merges into the output dir and does not depend on the _temporary/<appAttemptId> rename target

Example fix

// test / custom-committer code: ensure setupJob created the parent before v1 commit
// before
committer.commitTask(taskContext); // parent _temporary/<appAttemptId> may be absent

// after
committer.setupJob(jobContext);
committer.commitTask(taskContext);

// in tests or custom lifecycles: run setupJob before v1 commitTask so the rename parent exists
committer.setupJob(jobContext);
Path committedParent = committer.getCommittedTaskPath(taskContext).getParent();
if (!committedParent.getFileSystem(conf).exists(committedParent)) {
  throw new IOException("Committed-task parent missing: " + committedParent);
}

catch IOException from commitTask; extract taskAttemptPath/committedTaskPath from the message; check parent existence and perms (hadoop fs -ls <outdir>/_temporary) before retrying

When it happens

Trigger: Algorithm v1 commitTask where the target parent <outdir>/_temporary/<appAttemptId> does not exist (created by setupJob; missing if commitTask runs before setupJob, after abortJob/cleanupJob deleted _temporary, or when a JobContext for a different attempt is reused), or write permission on the parent is missing, or HDFS quota/disk is exhausted at rename time.

Common situations: Unit tests invoking commitTask directly without a prior setupJob; AM recovery flows where _temporary was cleaned; quota-full or NameNode-busy clusters; v1 committer on object stores with broken rename semantics.

Related errors

• Could not delete {}
• Cannot recover task output for first attempt...
• Unable to recover task %s, output: %s
• Changing job priority in LocalJobRunner is not supported.
• Killing tasks in LocalJobRunner is not supported

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/836da7a9a165c5d4. Report an issue: GitHub.