apache/hadoop · error · IOException

"Mkdirs failed to create " + reduceIn.getParent().toString()

Error message

"Mkdirs failed to create " + reduceIn.getParent().toString()

What it means

In an uberized MR job (mapreduce.job.ubertask.enable=true), map tasks run inside the AM's JVM and LocalContainerLauncher moves their outputs into reduce-input slots on the local filesystem. Before renaming, it calls localFs.mkdirs(reduceIn.getParent()) and throws IOException('Mkdirs failed to create <dir>') when the parent of the reduce input path cannot be created -- typically a full, read-only, or permission-denied local directory.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/LocalContainerLauncher.java:579

   * so there are no particular compatibility issues.)
   */
  @VisibleForTesting
  protected static MapOutputFile renameMapOutputForReduce(JobConf conf,
      TaskAttemptId mapId, MapOutputFile subMapOutputFile) throws IOException {
    FileSystem localFs = FileSystem.getLocal(conf);
    // move map output to reduce input
    Path mapOut = subMapOutputFile.getOutputFile();
    FileStatus mStatus = localFs.getFileStatus(mapOut);
    Path reduceIn = subMapOutputFile.getInputFileForWrite(
        TypeConverter.fromYarn(mapId).getTaskID(), mStatus.getLen());
    Path mapOutIndex = subMapOutputFile.getOutputIndexFile();
    Path reduceInIndex = new Path(reduceIn.toString() + ".index");
    if (LOG.isDebugEnabled()) {
      LOG.debug("Renaming map output file for task attempt {} from original location {}"
              + " to destination {}", mapId, mapOut, reduceIn);
    }
    if (!localFs.mkdirs(reduceIn.getParent())) {
      throw new IOException("Mkdirs failed to create "
          + reduceIn.getParent().toString());
    }
    if (!localFs.rename(mapOut, reduceIn))
      throw new IOException("Couldn't rename " + mapOut);
    if (!localFs.rename(mapOutIndex, reduceInIndex))
      throw new IOException("Couldn't rename " + mapOutIndex);

    return new RenamedMapOutputFile(reduceIn);
  }

  private static class RenamedMapOutputFile extends MapOutputFile {
    private Path path;
    
    public RenamedMapOutputFile(Path path) {
      this.path = path;
    }
    
    @Override

View on GitHub (pinned to 2add963021)

Solutions

1. On the node running the AM, check space and permissions of the mapreduce.cluster.local.dir volumes (df -h, ls -ld on the printed path)
2. Disable uber mode (mapreduce.job.ubertask.enable=false) so tasks run in normal NM containers with fresh localization
3. Remove stale attempt directories blocking the parent path, and correct ownership of usercache/<user> trees
4. Free disk or add local-dir capacity on that node

Example fix

<!-- mapred-site.xml: avoid AM-local task execution on disk-constrained nodes -->
<!-- before -->
<property><name>mapreduce.job.ubertask.enable</name><value>true</value></property>
<!-- after -->
<property><name>mapreduce.job.ubertask.enable</name><value>false</value></property>

# pre-flight on the AM node before submitting an uberized job
VOL=$(grep -A1 'mapreduce.cluster.local.dir' mapred-site.xml | tail -1 | sed 's/.*<value>//;s|<.*||' | cut -d, -f1)
df -h "$VOL" || exit 2
test -w "$VOL" || { echo "local dir not writable: $VOL" >&2; exit 2; }

Catch IOException around the task-execution block in uber-mode code and surface the failing path; treat mkdirs/rename failures as node-health problems (fail over to a non-uber run) rather than retrying on the same node.

When it happens

Trigger: Uber job on a node where the MR local dirs (mapreduce.cluster.local.dir) are full; the attempt output path's parent exists as a FILE left by a crashed earlier attempt; directory owned by another user so mkdirs returns false.

Common situations: Small jobs deliberately uberized to save containers, then failing on nodes with tight local disk; stale appcache/attempt directories after repeated failed runs; shared dev nodes with mixed ownership under local dirs.

Related errors

• "Couldn't rename " + mapOut
• "Couldn't rename " + mapOutIndex
• "Mkdirs failed to create " + workDir.toString()
• Mkdirs failed to create {} (exists={}, cwd={})
• Mkdirs failed to create ${badDir}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/18a55e496fd3a3fa. Report an issue: GitHub.