apache/hadoop · error · IOException

Unable to rename {src} to {dst}

Error message

Unable to rename {src} to {dst}

What it means

The single-spill fast path renames the spill file (and its index) to the final output files on the same volume using File.renameTo(). If the OS refuses the rename - destination already exists, permission denied, source vanished, or (on Windows) an open handle on the target - this IOException names both paths.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MapTask.java:2040

     * Rename srcPath to dstPath on the same volume. This is the same
     * as RawLocalFileSystem's rename method, except that it will not
     * fall back to a copy, and it will create the target directory
     * if it doesn't exist.
     */
    private void sameVolRename(Path srcPath,
        Path dstPath) throws IOException {
      RawLocalFileSystem rfs = (RawLocalFileSystem)this.rfs;
      File src = rfs.pathToFile(srcPath);
      File dst = rfs.pathToFile(dstPath);
      if (!dst.getParentFile().exists()) {
        if (!dst.getParentFile().mkdirs()) {
          throw new IOException("Unable to rename " + src + " to "
              + dst + ": couldn't create parent directory");
        }
      }
      
      if (!src.renameTo(dst)) {
        throw new IOException("Unable to rename " + src + " to " + dst);
      }
    }

    /**
     * Increments numSpills local counter by taking into consideration
     * the max limit on spill files being generated by the job.
     * If limit is reached, this function throws an IOException
     */
    private void incrementNumSpills() throws IOException {
      ++numSpills;
      if(spillFilesCountLimit != SPILL_FILES_COUNT_UNBOUNDED_LIMIT_VALUE
          && numSpills > spillFilesCountLimit) {
        throw new IOException("Too many spill files got created, control it with " +
            "mapreduce.task.spill.files.count.limit, current value: " + spillFilesCountLimit +
            ", current spill count: " + numSpills);
      }
    }
  } // MapOutputBuffer

View on GitHub (pinned to 2add963021)

Solutions

1. Inspect both paths from the message on the node: does the destination already exist? does the source still exist?
2. Clear stale attempt directories under the NM local dirs and rerun the job
3. Verify the attempt directory is owned by the task user and writable
4. On Windows nodes, exclude Hadoop local dirs from antivirus scanning

When it happens

Trigger: File.renameTo returns false during mergeParts' numSpills==1 promotion: stale output file left from prior attempt residue in the attempt dir, source spill file already deleted by cleanup races, EACCES on the directory, or an open handle on Windows.

Common situations: Leftover files in attempt output dirs after NM restarts or interrupted attempts; antivirus/backup software holding handles on Windows; unwritable directory ownership; source removed mid-finalization.

Related errors

• Unable to rename {src} to {dst}: couldn't create parent dire
• "Couldn't rename " + mapOut
• "Couldn't rename " + mapOutIndex
• Spill thread failed to initialize
• Buffer interrupted while waiting for the writer

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/143b090ae4490e00. Report an issue: GitHub.