{"record":{"id":"836da7a9a165c5d4","repo":"apache/hadoop","slug":"could-not-rename-to","errorCode":null,"errorMessage":"Could not rename {} to {}","messagePattern":"Could not rename (.+?) to (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter.java","lineNumber":606,"sourceCode":"      }\n      FileSystem fs = taskAttemptPath.getFileSystem(context.getConfiguration());\n      FileStatus taskAttemptDirStatus;\n      try {\n        taskAttemptDirStatus = fs.getFileStatus(taskAttemptPath);\n      } catch (FileNotFoundException e) {\n        taskAttemptDirStatus = null;\n      }\n\n      if (taskAttemptDirStatus != null) {\n        if (algorithmVersion == 1) {\n          Path committedTaskPath = getCommittedTaskPath(context);\n          if (fs.exists(committedTaskPath)) {\n             if (!fs.delete(committedTaskPath, true)) {\n               throw new IOException(\"Could not delete \" + committedTaskPath);\n             }\n          }\n          if (!fs.rename(taskAttemptPath, committedTaskPath)) {\n            throw new IOException(\"Could not rename \" + taskAttemptPath + \" to \"\n                + committedTaskPath);\n          }\n          LOG.info(\"Saved output of task '\" + attemptId + \"' to \" +\n              committedTaskPath);\n        } else {\n          // directly merge everything from taskAttemptPath to output directory\n          mergePaths(fs, taskAttemptDirStatus, outputPath, context);\n          LOG.info(\"Saved output of task '\" + attemptId + \"' to \" +\n              outputPath);\n\n          if (context.getConfiguration().getBoolean(\n              FILEOUTPUTCOMMITTER_TASK_CLEANUP_ENABLED,\n              FILEOUTPUTCOMMITTER_TASK_CLEANUP_ENABLED_DEFAULT)) {\n            LOG.debug(String.format(\n                \"Deleting the temporary directory of '%s': '%s'\",\n                attemptId, taskAttemptPath));\n            if(!fs.delete(taskAttemptPath, true)) {\n              LOG.warn(\"Could not delete \" + taskAttemptPath);","sourceCodeStart":588,"sourceCodeEnd":624,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter.java#L588-L624","documentation":"Thrown from FileOutputCommitter.commitTask (FileOutputCommitter.java:606) under commit algorithm v1 after the (optional) delete of an old committed-task dir succeeded but the subsequent fs.rename(taskAttemptPath, committedTaskPath) returned false. This is the actual move of task output from <outdir>/_temporary/<appAttemptId>/attempt_*/ into the committed-task slot; a false return means the FS rejected the rename.","triggerScenarios":"Algorithm v1 commitTask where the target parent <outdir>/_temporary/<appAttemptId> does not exist (created by setupJob; missing if commitTask runs before setupJob, after abortJob/cleanupJob deleted _temporary, or when a JobContext for a different attempt is reused), or write permission on the parent is missing, or HDFS quota/disk is exhausted at rename time.","commonSituations":"Unit tests invoking commitTask directly without a prior setupJob; AM recovery flows where _temporary was cleaned; quota-full or NameNode-busy clusters; v1 committer on object stores with broken rename semantics.","solutions":["Confirm <outdir>/_temporary/<appAttemptId> exists at commit time; if it was deleted mid-job, rerun into a fresh output directory","Verify write permission on the _temporary tree for the job user and fix ownership (hadoop fs -chown -R)","Check HDFS quotas/disk (hadoop fs -count -q <outdir>) and free space","Move to mapreduce.fileoutputcommitter.algorithm.version=2, which merges into the output dir and does not depend on the _temporary/<appAttemptId> rename target"],"exampleFix":"// test / custom-committer code: ensure setupJob created the parent before v1 commit\n// before\ncommitter.commitTask(taskContext); // parent _temporary/<appAttemptId> may be absent\n\n// after\ncommitter.setupJob(jobContext);\ncommitter.commitTask(taskContext);","handlingStrategy":"validation","validationCode":"// in tests or custom lifecycles: run setupJob before v1 commitTask so the rename parent exists\ncommitter.setupJob(jobContext);\nPath committedParent = committer.getCommittedTaskPath(taskContext).getParent();\nif (!committedParent.getFileSystem(conf).exists(committedParent)) {\n  throw new IOException(\"Committed-task parent missing: \" + committedParent);\n}","typeGuard":null,"tryCatchPattern":"catch IOException from commitTask; extract taskAttemptPath/committedTaskPath from the message; check parent existence and perms (hadoop fs -ls <outdir>/_temporary) before retrying","preventionTips":["Always let the framework drive setupJob before task commits","Monitor cluster quota/disk before large job fleets","Do not clean _temporary mid-job"],"tags":["hadoop","mapreduce","file-output-committer","hdfs-rename","task-commit","algorithm-v1"],"backgroundTag":"mapreduce-output-commit-failed","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}