apache/hadoop · error · IOException

"Mkdirs failed to create " + workDir.toString()

Error message

"Mkdirs failed to create " + workDir.toString()

What it means

At task startup YarnChild creates the task's work directory inside the localized job directory. It calls lfs.mkdirs(workDir); a FileAlreadyExistsException is tolerated (race with sibling tasks), but a plain false return becomes IOException('Mkdirs failed to create <workDir>'). False means the local filesystem refused to create the directory: no space, read-only volume, permission denied, or a regular file occupying a path component.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/YarnChild.java:316

      // be created below.
    }
    if (workDir == null) {
      // JOB_LOCAL_DIR doesn't exist on this host -- Create it.
      workDir = lDirAlloc.getLocalPathForWrite("work", job);
      FileSystem lfs = FileSystem.getLocal(job).getRaw();
      boolean madeDir = false;
      try {
        madeDir = lfs.mkdirs(workDir);
      } catch (FileAlreadyExistsException e) {
        // Since all tasks will be running in their own JVM, the race condition
        // exists where multiple tasks could be trying to create this directory
        // at the same time. If this task loses the race, it's okay because
        // the directory already exists.
        madeDir = true;
        workDir = lDirAlloc.getLocalPathToRead("work", job);
      }
      if (!madeDir) {
          throw new IOException("Mkdirs failed to create "
              + workDir.toString());
      }
    }
    job.set(MRJobConfig.JOB_LOCAL_DIR,workDir.toString());
  }

  private static void configureTask(JobConf job, Task task,
      Credentials credentials, Token<JobTokenIdentifier> jt) throws IOException {
    job.setCredentials(credentials);

    ApplicationAttemptId appAttemptId = ContainerId.fromString(
        System.getenv(Environment.CONTAINER_ID.name()))
        .getApplicationAttemptId();
    LOG.debug("APPLICATION_ATTEMPT_ID: {}", appAttemptId);
    // Set it in conf, so as to be able to be used the the OutputCommitter.
    job.setInt(MRJobConfig.APPLICATION_ATTEMPT_ID,
        appAttemptId.getAttemptId());

View on GitHub (pinned to 2add963021)

Solutions

1. On the NodeManager host, inspect the exact path from the message: df -h <vol>, ls -ld each component, confirm no regular file blocks the path
2. Free space on the local-dir volume or add capacity, then let the attempt relocalize
3. chown the usercache/<user> tree to the submitting user (or purge stale appcache dirs) so mkdirs can succeed
4. If a single node is unhealthy, blacklist it via yarn.resourcemanager.node-blacklisting-enabled or NM health scripts

Example fix

# inspect and unblock the path from the error message on the NM host
ls -ld /yarn/local/usercache/alice/appcache/application_123
# remove a blocking file / fix ownership, then resubmit
rm -f /yarn/local/usercache/alice/appcache/application_123/work
chown -R alice:hadoop /yarn/local/usercache/alice

# node health check for NM hosts: writable local dirs with free space
for d in $(yarn node -status $NODEID 2>/dev/null; echo /yarn/local); do :; done
# simplest guard: NM health-check script asserting writability
# health_check.sh: touch /yarn/local/.w && rm /yarn/local/.w || exit 1

Catch IOException in task-launch wrappers and read the printed workDir path; classify as node-local (disk/permissions) -- fix the node or blacklist it, then resubmit rather than looping retries on the same host.

When it happens

Trigger: Task attempt scheduled on a node whose local dir volume is 100% full; work dir parent exists as a file left by a crashed attempt; dir owned by another user after uid changes; disk mounted read-only after hardware trouble.

Common situations: NodeManager local disks (yarn.nodemanager.local.dirs / mapreduce.cluster.local.dir) full or read-only on that node; leftover file where the work dir should be; usercache/<user> owned by a different uid after user re-creation; disk full specifically in the user's appcache after many failed attempts.

Related errors

• "Mkdirs failed to create " + reduceIn.getParent().toString()
• Unable to rename {src} to {dst}: couldn't create parent dire
• Mkdirs failed to create {} (exists={}, cwd={})
• Mkdirs failed to create " + parent.getAbsolutePath()
• Mkdirs failed to create " + file.getParentFile().toString()

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/fdf4cf97fec7ded5. Report an issue: GitHub.