apache/hadoop · error · IOException

"Couldn't rename " + mapOutIndex

Error message

"Couldn't rename " + mapOutIndex

What it means

Companion to the map-output rename in LocalContainerLauncher: after successfully renaming the map output data file, the code renames the matching spill index file (mapOutIndex -> reduceIn + '.index'). If RawLocalFileSystem.rename returns false for that second file, the job throws IOException('Couldn't rename <mapOutIndex>') and the map attempt fails. Because the data file was already moved, the slot is now half-populated and a rerun needs cleanup.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/LocalContainerLauncher.java:585

    // move map output to reduce input
    Path mapOut = subMapOutputFile.getOutputFile();
    FileStatus mStatus = localFs.getFileStatus(mapOut);
    Path reduceIn = subMapOutputFile.getInputFileForWrite(
        TypeConverter.fromYarn(mapId).getTaskID(), mStatus.getLen());
    Path mapOutIndex = subMapOutputFile.getOutputIndexFile();
    Path reduceInIndex = new Path(reduceIn.toString() + ".index");
    if (LOG.isDebugEnabled()) {
      LOG.debug("Renaming map output file for task attempt {} from original location {}"
              + " to destination {}", mapId, mapOut, reduceIn);
    }
    if (!localFs.mkdirs(reduceIn.getParent())) {
      throw new IOException("Mkdirs failed to create "
          + reduceIn.getParent().toString());
    }
    if (!localFs.rename(mapOut, reduceIn))
      throw new IOException("Couldn't rename " + mapOut);
    if (!localFs.rename(mapOutIndex, reduceInIndex))
      throw new IOException("Couldn't rename " + mapOutIndex);

    return new RenamedMapOutputFile(reduceIn);
  }

  private static class RenamedMapOutputFile extends MapOutputFile {
    private Path path;
    
    public RenamedMapOutputFile(Path path) {
      this.path = path;
    }
    
    @Override
    public Path getOutputFile() throws IOException {
      return path;
    }

    @Override
    public Path getOutputFileForWrite(long size) throws IOException {

View on GitHub (pinned to 2add963021)

Solutions

1. Inspect the exact mapOutIndex path from the message and its destination; delete the leftover reduceIn and reduceIn.index pair from the failed attempt before retrying
2. Fix ownership/permissions so both .out and .out.index share the same writable directory
3. Exclude MR local dirs from janitor/antivirus scans, or move mapreduce.cluster.local.dir out of cleaned temp paths
4. If it recurs, disable uber mode so tasks get isolated, freshly localized containers

Example fix

# cleanup of a half-moved slot before rerunning the job (paths from the error message)
# before: failed attempt left reduce input pair inconsistent
ls -l /mrlocal/usercache/alice/appcache/application_123/output/attempt_*
# after: remove the stale pair and resubmit
rm -rf /mrlocal/usercache/alice/appcache/application_123/output/attempt_<failed>
yarn jar myjob.jar -Dmapreduce.job.ubertask.enable=false ...

Catch IOException on the index-file rename and, because the data file already moved, remove BOTH reduceIn and reduceIn.index before the attempt is retried -- otherwise the next run fails on the stale pair.

When it happens

Trigger: Stale '.index' destination file from an earlier attempt; index file removed by an aggressive cleanup thread mid-rename; ownership/permission differing between the .out and .out.index files; cross-volume allocation of the pair.

Common situations: Retried uber attempts after disk-full events leave partial reduce-input slots; antivirus or tmpwatch-style janitors deleting .index files under local dirs; mixed ownership under usercache.

Related errors

• "Couldn't rename " + mapOut
• "Mkdirs failed to create " + reduceIn.getParent().toString()
• Unable to rename {src} to {dst}
• Failed to rename %s to %s
• Not yet implemented.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/d84c377419c9a90a. Report an issue: GitHub.