apache/hadoop · error · IOException

"Couldn't rename " + mapOut

Error message

"Couldn't rename " + mapOut

What it means

In uber mode, LocalContainerLauncher renames each map output file into the reduce input slot on the local filesystem. RawLocalFileSystem.rename returns false (rather than throwing) for a cross-volume rename, a missing source, or insufficient permission, and each false becomes IOException('Couldn't rename <mapOut>'). The path in the message is the map attempt's spill/output file under the task's attempt output directory.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/LocalContainerLauncher.java:583

      TaskAttemptId mapId, MapOutputFile subMapOutputFile) throws IOException {
    FileSystem localFs = FileSystem.getLocal(conf);
    // move map output to reduce input
    Path mapOut = subMapOutputFile.getOutputFile();
    FileStatus mStatus = localFs.getFileStatus(mapOut);
    Path reduceIn = subMapOutputFile.getInputFileForWrite(
        TypeConverter.fromYarn(mapId).getTaskID(), mStatus.getLen());
    Path mapOutIndex = subMapOutputFile.getOutputIndexFile();
    Path reduceInIndex = new Path(reduceIn.toString() + ".index");
    if (LOG.isDebugEnabled()) {
      LOG.debug("Renaming map output file for task attempt {} from original location {}"
              + " to destination {}", mapId, mapOut, reduceIn);
    }
    if (!localFs.mkdirs(reduceIn.getParent())) {
      throw new IOException("Mkdirs failed to create "
          + reduceIn.getParent().toString());
    }
    if (!localFs.rename(mapOut, reduceIn))
      throw new IOException("Couldn't rename " + mapOut);
    if (!localFs.rename(mapOutIndex, reduceInIndex))
      throw new IOException("Couldn't rename " + mapOutIndex);

    return new RenamedMapOutputFile(reduceIn);
  }

  private static class RenamedMapOutputFile extends MapOutputFile {
    private Path path;
    
    public RenamedMapOutputFile(Path path) {
      this.path = path;
    }
    
    @Override
    public Path getOutputFile() throws IOException {
      return path;
    }

View on GitHub (pinned to 2add963021)

Solutions

1. Check the two paths named in the log (source mapOut and destination reduceIn) with ls -l on the AM node; confirm same volume, ownership, and that no destination file exists
2. Clean stale attempt directories under the app's output area, then rerun the job
3. Force spill and reduce-input files onto one volume by providing a single writable mapreduce.cluster.local.dir for these jobs, or disable uber mode
4. Verify the AM user has write permission on every local-dir volume

Example fix

# before: two volumes, spill and reduce-input may split across them
<property><name>mapreduce.cluster.local.dir</name><value>/disk0/mrlocal,/disk1/mrlocal</value></property>
# after: single volume for the uberized job
<property><name>mapreduce.cluster.local.dir</name><value>/disk0/mrlocal</value></property>

# verify same-volume placement and clean destination before an uber job
# (paths appear in the AM log line 'Renaming map output file ...')

Catch IOException from the rename step, log both source and destination paths, and abort the attempt after cleaning the partial destination; do not blindly retry -- a false-returning raw rename usually means EXDEV, permissions, or an existing destination.

When it happens

Trigger: Map output and reduce input land on different local-dir volumes (EXDEV); the destination already exists from a previous failed attempt; the attempt dir is owned by another user; the map output file was already moved or deleted.

Common situations: Uber jobs on nodes with multiple mapreduce.cluster.local.dir volumes where the dir allocator splits spill and reduce-input paths; leftover files from retried attempts; restrictive umask/ownership under usercache.

Related errors

• "Couldn't rename " + mapOutIndex
• "Mkdirs failed to create " + reduceIn.getParent().toString()
• Unable to rename {src} to {dst}
• Failed to rename %s to %s
• Not yet implemented.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/344ed011a9682645. Report an issue: GitHub.