{"record":{"id":"7476b86703506a33","repo":"apache/hadoop","slug":"could-not-delete","errorCode":null,"errorMessage":"Could not delete {}","messagePattern":"Could not delete (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter.java","lineNumber":602,"sourceCode":"    if (hasOutputPath()) {\n      context.progress();\n      if(taskAttemptPath == null) {\n        taskAttemptPath = getTaskAttemptPath(context);\n      }\n      FileSystem fs = taskAttemptPath.getFileSystem(context.getConfiguration());\n      FileStatus taskAttemptDirStatus;\n      try {\n        taskAttemptDirStatus = fs.getFileStatus(taskAttemptPath);\n      } catch (FileNotFoundException e) {\n        taskAttemptDirStatus = null;\n      }\n\n      if (taskAttemptDirStatus != null) {\n        if (algorithmVersion == 1) {\n          Path committedTaskPath = getCommittedTaskPath(context);\n          if (fs.exists(committedTaskPath)) {\n             if (!fs.delete(committedTaskPath, true)) {\n               throw new IOException(\"Could not delete \" + committedTaskPath);\n             }\n          }\n          if (!fs.rename(taskAttemptPath, committedTaskPath)) {\n            throw new IOException(\"Could not rename \" + taskAttemptPath + \" to \"\n                + committedTaskPath);\n          }\n          LOG.info(\"Saved output of task '\" + attemptId + \"' to \" +\n              committedTaskPath);\n        } else {\n          // directly merge everything from taskAttemptPath to output directory\n          mergePaths(fs, taskAttemptDirStatus, outputPath, context);\n          LOG.info(\"Saved output of task '\" + attemptId + \"' to \" +\n              outputPath);\n\n          if (context.getConfiguration().getBoolean(\n              FILEOUTPUTCOMMITTER_TASK_CLEANUP_ENABLED,\n              FILEOUTPUTCOMMITTER_TASK_CLEANUP_ENABLED_DEFAULT)) {\n            LOG.debug(String.format(","sourceCodeStart":584,"sourceCodeEnd":620,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter.java#L584-L620","documentation":"Thrown from FileOutputCommitter.commitTask (FileOutputCommitter.java:602) under commit algorithm v1. Before renaming the task-attempt directory into its committed-task path, v1 first deletes any committed-task directory left by a previous attempt of the same task. If fs.delete(committedTaskPath, true) returns false while the path is known to exist (fs.exists() was just checked), this IOException fails the task commit.","triggerScenarios":"mapreduce.fileoutputcommitter.algorithm.version=1, commitTask() called for a retried task attempt whose committed-task path already exists under <outdir>/_temporary/<appAttemptId>/task_*/, and fs.delete returns false. Concrete causes: no delete permission on that committed directory (often created by a different user in a prior run), a concurrent commit of another attempt of the same task, or FS-level inconsistency.","commonSituations":"Task retries after a failed attempt under v1; output directory reused across runs by different users so committed-task dirs are owned by someone else; security-hardened clusters where the job user can create but not delete sibling files.","solutions":["Check ownership/permissions of the committed-task path printed in the message (hadoop fs -ls <outdir>/_temporary) and hadoop fs -rm -r it as an admin or the owning user, then let the task retry","Run each job into a fresh output directory so no committed-task dirs from previous runs exist","Ensure the same user (and only that user) runs all attempts of the job","Switch to mapreduce.fileoutputcommitter.algorithm.version=2 (default in modern Hadoop), which merges directly into the output dir and does not need this delete"],"exampleFix":"// job setup: stop reusing the v1 commit path layout across runs\n// before\nconf.setInt(\"mapreduce.fileoutputcommitter.algorithm.version\", 1);\nFileOutputFormat.setOutputPath(job, new Path(\"/data/shared-out\"));\n\n// after\nconf.setInt(\"mapreduce.fileoutputcommitter.algorithm.version\", 2);\nFileOutputFormat.setOutputPath(job, new Path(\"/data/out-\" + job.getJobName()));","handlingStrategy":"validation","validationCode":"// pre-submit sanity: the output tree must be owned by / deletable by the job user\nFileSystem fs = outDir.getFileSystem(conf);\nif (fs.exists(outDir)) {\n  FileStatus st = fs.getFileStatus(outDir);\n  FsShellPermissionCheck: verify st.getOwner().equals(currentUser) or writable bits;\n}","typeGuard":null,"tryCatchPattern":"try { committer.commitTask(context); } catch (IOException io) { if (io.getMessage().contains(\"Could not delete\")) { /* inspect committed-task dir ownership; admin delete + allow task retry */ } throw io; }","preventionTips":["Fresh output dir per run","Single owning user for the whole job lifecycle","Avoid mixing v1 committer with reused output directories"],"tags":["hadoop","mapreduce","file-output-committer","hdfs-delete","task-commit","algorithm-v1","task-retry"],"backgroundTag":"mapreduce-output-commit-failed","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}