{"record":{"id":"fdf4cf97fec7ded5","repo":"apache/hadoop","slug":"mkdirs-failed-to-create-workdir-tostring","errorCode":null,"errorMessage":"\"Mkdirs failed to create \" + workDir.toString()","messagePattern":"\"Mkdirs failed to create \" \\+ workDir\\.toString\\(\\)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/YarnChild.java","lineNumber":316,"sourceCode":"      // be created below.\n    }\n    if (workDir == null) {\n      // JOB_LOCAL_DIR doesn't exist on this host -- Create it.\n      workDir = lDirAlloc.getLocalPathForWrite(\"work\", job);\n      FileSystem lfs = FileSystem.getLocal(job).getRaw();\n      boolean madeDir = false;\n      try {\n        madeDir = lfs.mkdirs(workDir);\n      } catch (FileAlreadyExistsException e) {\n        // Since all tasks will be running in their own JVM, the race condition\n        // exists where multiple tasks could be trying to create this directory\n        // at the same time. If this task loses the race, it's okay because\n        // the directory already exists.\n        madeDir = true;\n        workDir = lDirAlloc.getLocalPathToRead(\"work\", job);\n      }\n      if (!madeDir) {\n          throw new IOException(\"Mkdirs failed to create \"\n              + workDir.toString());\n      }\n    }\n    job.set(MRJobConfig.JOB_LOCAL_DIR,workDir.toString());\n  }\n\n  private static void configureTask(JobConf job, Task task,\n      Credentials credentials, Token<JobTokenIdentifier> jt) throws IOException {\n    job.setCredentials(credentials);\n\n    ApplicationAttemptId appAttemptId = ContainerId.fromString(\n        System.getenv(Environment.CONTAINER_ID.name()))\n        .getApplicationAttemptId();\n    LOG.debug(\"APPLICATION_ATTEMPT_ID: {}\", appAttemptId);\n    // Set it in conf, so as to be able to be used the the OutputCommitter.\n    job.setInt(MRJobConfig.APPLICATION_ATTEMPT_ID,\n        appAttemptId.getAttemptId());\n","sourceCodeStart":298,"sourceCodeEnd":334,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/YarnChild.java#L298-L334","documentation":"At task startup YarnChild creates the task's work directory inside the localized job directory. It calls lfs.mkdirs(workDir); a FileAlreadyExistsException is tolerated (race with sibling tasks), but a plain false return becomes IOException('Mkdirs failed to create <workDir>'). False means the local filesystem refused to create the directory: no space, read-only volume, permission denied, or a regular file occupying a path component.","triggerScenarios":"Task attempt scheduled on a node whose local dir volume is 100% full; work dir parent exists as a file left by a crashed attempt; dir owned by another user after uid changes; disk mounted read-only after hardware trouble.","commonSituations":"NodeManager local disks (yarn.nodemanager.local.dirs / mapreduce.cluster.local.dir) full or read-only on that node; leftover file where the work dir should be; usercache/<user> owned by a different uid after user re-creation; disk full specifically in the user's appcache after many failed attempts.","solutions":["On the NodeManager host, inspect the exact path from the message: df -h <vol>, ls -ld each component, confirm no regular file blocks the path","Free space on the local-dir volume or add capacity, then let the attempt relocalize","chown the usercache/<user> tree to the submitting user (or purge stale appcache dirs) so mkdirs can succeed","If a single node is unhealthy, blacklist it via yarn.resourcemanager.node-blacklisting-enabled or NM health scripts"],"exampleFix":"# inspect and unblock the path from the error message on the NM host\nls -ld /yarn/local/usercache/alice/appcache/application_123\n# remove a blocking file / fix ownership, then resubmit\nrm -f /yarn/local/usercache/alice/appcache/application_123/work\nchown -R alice:hadoop /yarn/local/usercache/alice","handlingStrategy":"try-catch","validationCode":"# node health check for NM hosts: writable local dirs with free space\nfor d in $(yarn node -status $NODEID 2>/dev/null; echo /yarn/local); do :; done\n# simplest guard: NM health-check script asserting writability\n# health_check.sh: touch /yarn/local/.w && rm /yarn/local/.w || exit 1","typeGuard":null,"tryCatchPattern":"Catch IOException in task-launch wrappers and read the printed workDir path; classify as node-local (disk/permissions) -- fix the node or blacklist it, then resubmit rather than looping retries on the same host.","preventionTips":["Configure NodeManager health-check scripts that test local-dir writability and free space","Regularly purge stale usercache/appcache directories for departed users","Alert on local-dir disk utilization above ~85% on NodeManagers"],"tags":["mapreduce","yarn","node-manager","local-dirs","mkdir","disk-full","permissions"],"backgroundTag":"cannot-create-directory","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}