apache/hadoop · error · IOException

Could not delete

Error message

Could not delete {}

What it means

Thrown from FileOutputCommitter.commitTask (FileOutputCommitter.java:602) under commit algorithm v1. Before renaming the task-attempt directory into its committed-task path, v1 first deletes any committed-task directory left by a previous attempt of the same task. If fs.delete(committedTaskPath, true) returns false while the path is known to exist (fs.exists() was just checked), this IOException fails the task commit.

Solutions

1. Check ownership/permissions of the committed-task path printed in the message (hadoop fs -ls <outdir>/_temporary) and hadoop fs -rm -r it as an admin or the owning user, then let the task retry
2. Run each job into a fresh output directory so no committed-task dirs from previous runs exist
3. Ensure the same user (and only that user) runs all attempts of the job
4. Switch to mapreduce.fileoutputcommitter.algorithm.version=2 (default in modern Hadoop), which merges directly into the output dir and does not need this delete

Example fix

// job setup: stop reusing the v1 commit path layout across runs
// before
conf.setInt("mapreduce.fileoutputcommitter.algorithm.version", 1);
FileOutputFormat.setOutputPath(job, new Path("/data/shared-out"));

// after
conf.setInt("mapreduce.fileoutputcommitter.algorithm.version", 2);
FileOutputFormat.setOutputPath(job, new Path("/data/out-" + job.getJobName()));

// pre-submit sanity: the output tree must be owned by / deletable by the job user
FileSystem fs = outDir.getFileSystem(conf);
if (fs.exists(outDir)) {
  FileStatus st = fs.getFileStatus(outDir);
  FsShellPermissionCheck: verify st.getOwner().equals(currentUser) or writable bits;
}

try { committer.commitTask(context); } catch (IOException io) { if (io.getMessage().contains("Could not delete")) { /* inspect committed-task dir ownership; admin delete + allow task retry */ } throw io; }

When it happens

Trigger: mapreduce.fileoutputcommitter.algorithm.version=1, commitTask() called for a retried task attempt whose committed-task path already exists under <outdir>/_temporary/<appAttemptId>/task_*/, and fs.delete returns false. Concrete causes: no delete permission on that committed directory (often created by a different user in a prior run), a concurrent commit of another attempt of the same task, or FS-level inconsistency.

Common situations: Task retries after a failed attempt under v1; output directory reused across runs by different users so committed-task dirs are owned by someone else; security-hardened clusters where the job user can create but not delete sibling files.

Related errors

• Could not rename to
• Cannot recover task output for first attempt...
• A Mapper can be added to the chain only after the Reducer…
• Attempt to remove non-existent val
• bad framework group id

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/7476b86703506a33. Report an issue: GitHub.

Appendix: source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter.java:602

    if (hasOutputPath()) {
      context.progress();
      if(taskAttemptPath == null) {
        taskAttemptPath = getTaskAttemptPath(context);
      }
      FileSystem fs = taskAttemptPath.getFileSystem(context.getConfiguration());
      FileStatus taskAttemptDirStatus;
      try {
        taskAttemptDirStatus = fs.getFileStatus(taskAttemptPath);
      } catch (FileNotFoundException e) {
        taskAttemptDirStatus = null;
      }

      if (taskAttemptDirStatus != null) {
        if (algorithmVersion == 1) {
          Path committedTaskPath = getCommittedTaskPath(context);
          if (fs.exists(committedTaskPath)) {
             if (!fs.delete(committedTaskPath, true)) {
               throw new IOException("Could not delete " + committedTaskPath);
             }
          }
          if (!fs.rename(taskAttemptPath, committedTaskPath)) {
            throw new IOException("Could not rename " + taskAttemptPath + " to "
                + committedTaskPath);
          }
          LOG.info("Saved output of task '" + attemptId + "' to " +
              committedTaskPath);
        } else {
          // directly merge everything from taskAttemptPath to output directory
          mergePaths(fs, taskAttemptDirStatus, outputPath, context);
          LOG.info("Saved output of task '" + attemptId + "' to " +
              outputPath);

          if (context.getConfiguration().getBoolean(
              FILEOUTPUTCOMMITTER_TASK_CLEANUP_ENABLED,
              FILEOUTPUTCOMMITTER_TASK_CLEANUP_ENABLED_DEFAULT)) {
            LOG.debug(String.format(

View on GitHub (pinned to 2add963021)