What causes "Failed to delete {}" in apache/hadoop?

Clean the output directory before rerunning: hadoop fs -rm -r /output (or write to a new output path per run)

apache/hadoop · critical · IOException

Failed to delete {}

Error message

Failed to delete {}

What it means

During commit, FileOutputCommitter.mergePaths (FileOutputCommitter.java:461+) moves task-attempt output into the final output directory. When the destination path already exists, it first deletes it with fs.delete(to, true); if the FileSystem returns false (delete failed without throwing), the committer throws IOException('Failed to delete <to>'). This is a commit-phase failure, typically surfaced in task/AM commit or job-commit logs.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter.java:477

   * @throws IOException on any error
   */
  private void mergePaths(FileSystem fs, final FileStatus from,
      final Path to, JobContext context) throws IOException {
    try (DurationInfo d = new DurationInfo(LOG,
        false,
        "Merging data from %s to %s", from, to)) {
      reportProgress(context);
      FileStatus toStat;
      try {
        toStat = fs.getFileStatus(to);
      } catch (FileNotFoundException fnfe) {
        toStat = null;
      }

      if (from.isFile()) {
        if (toStat != null) {
          if (!fs.delete(to, true)) {
            throw new IOException("Failed to delete " + to);
          }
        }

        if (!fs.rename(from.getPath(), to)) {
          throw new IOException("Failed to rename " + from + " to " + to);
        }
      } else if (from.isDirectory()) {
        if (toStat != null) {
          if (!toStat.isDirectory()) {
            if (!fs.delete(to, true)) {
              throw new IOException("Failed to delete " + to);
            }
            renameOrMerge(fs, from, to, context);
          } else {
            //It is a directory so merge everything in the directories
            for (FileStatus subFrom : fs.listStatus(from.getPath())) {
              Path subTo = new Path(to, subFrom.getPath().getName());
              mergePaths(fs, subFrom, subTo, context);

View on GitHub (pinned to 2add963021)

Solutions

Clean the output directory before rerunning: hadoop fs -rm -r /output (or write to a new output path per run)
Verify write+delete permissions on the output directory for the running user (hadoop fs -ls, -touchz a scratch file then delete it)
Prevent concurrent jobs from sharing an output path; add unique run IDs/staging dirs to output locations
Check fs health at failure time: HDFS safe mode (hdfs dfsadmin -safemode get), NameNode failover mid-commit; retry the job after the cluster is healthy
For S3 outputs, configure a real S3A committer (magic or directory committer via fs.s3a.committer.name) instead of relying on FileOutputCommitter renames

Example fix

# before
hadoop jar app.jar Driver /in /out   # /out still contains committed files from previous failed run

# after
hadoop fs -rm -r /out/_temporary 2>/dev/null
hadoop fs -rm -r /out               # or: /out-$(date +%s) per-run output
hadoop jar app.jar Driver /in /out

Defensive patterns

Strategy: retry

Validate before calling

static void requireCleanWritableOutput(Path out, Configuration conf) throws IOException {
  FileSystem fs = out.getFileSystem(conf);
  if (fs.exists(out)) {
    FileStatus st = fs.getFileStatus(out);
    if (!st.getPermission().getUserAction().implies(FsAction.WRITE))
      throw new IllegalStateException("Output dir not writable: " + out);
    if (fs.exists(new Path(out, "_temporary")))
      throw new IllegalStateException("Stale _temporary present; remove it or use a fresh output dir");
  }
}

Try / catch

catch (IOException e) { if (String.valueOf(e.getMessage()).startsWith("Failed to delete")) { // transient FS states (safe mode, failover) often clear: clean output, verify permissions, retry job once
  throw new IOException("Commit-time delete failed for " + e.getMessage() + " — clean output dir, check permissions/safe-mode, rerun", e); } throw e; }

Prevention

Always write to a fresh (or explicitly cleaned) output directory per run
Ensure the submitting user can delete under the output path (check perms + owner)
Never point concurrent jobs at the same output directory
For object stores use store-native committers instead of rename/delete-based commit

When it happens

Trigger: Destination file/dir already exists in the output directory and cannot be deleted: permissions lacking on the parent, read-only or safe-mode HDFS, another job concurrently writing the same output path, a stale _temporary directory from a previous failed attempt containing the same names, or object-store semantics (S3A) where delete/rename behave differently.

Common situations: Rerunning jobs into an existing output directory that contains leftovers; two concurrent jobs sharing one output path; speculative tasks or AM retry racing on commit; misconfigured permissions on the output dir; using algorithm version 1 vs 2 with shared output locations; S3 without a proper S3A committer configured.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/14f1cd99687949ec. Report an issue: GitHub.