{"record":{"id":"7f08a0983b480059","repo":"apache/hadoop","slug":"unable-to-rename-src-to-dst-couldn-t-create-p","errorCode":null,"errorMessage":"Unable to rename {src} to {dst}: couldn't create parent directory","messagePattern":"Unable to rename (.+?) to (.+?): couldn't create parent directory","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MapTask.java","lineNumber":2034,"sourceCode":"          rfs.delete(filename[i],true);\n        }\n      }\n    }\n    \n    /**\n     * Rename srcPath to dstPath on the same volume. This is the same\n     * as RawLocalFileSystem's rename method, except that it will not\n     * fall back to a copy, and it will create the target directory\n     * if it doesn't exist.\n     */\n    private void sameVolRename(Path srcPath,\n        Path dstPath) throws IOException {\n      RawLocalFileSystem rfs = (RawLocalFileSystem)this.rfs;\n      File src = rfs.pathToFile(srcPath);\n      File dst = rfs.pathToFile(dstPath);\n      if (!dst.getParentFile().exists()) {\n        if (!dst.getParentFile().mkdirs()) {\n          throw new IOException(\"Unable to rename \" + src + \" to \"\n              + dst + \": couldn't create parent directory\");\n        }\n      }\n      \n      if (!src.renameTo(dst)) {\n        throw new IOException(\"Unable to rename \" + src + \" to \" + dst);\n      }\n    }\n\n    /**\n     * Increments numSpills local counter by taking into consideration\n     * the max limit on spill files being generated by the job.\n     * If limit is reached, this function throws an IOException\n     */\n    private void incrementNumSpills() throws IOException {\n      ++numSpills;\n      if(spillFilesCountLimit != SPILL_FILES_COUNT_UNBOUNDED_LIMIT_VALUE\n          && numSpills > spillFilesCountLimit) {","sourceCodeStart":2016,"sourceCodeEnd":2052,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MapTask.java#L2016-L2052","documentation":"During map finalization, when exactly one spill file was produced, mergeParts() promotes that spill (and its index) to the final map output by renaming within the same volume. sameVolRename first ensures the destination's parent directory exists; if the parent is missing and mkdirs() fails - permissions, read-only mount, disk full, or a deleted attempt directory - this IOException is thrown.","triggerScenarios":"The numSpills == 1 fast path calls getOutputFileForWriteInVolume(filename[0]) and renames the spill into place; the target parent directory cannot be created because the NM local dir is full, unwritable, read-only after hardware trouble, or the attempt dir was removed underneath the task.","commonSituations":"Local-disk capacity or permission problems on the NodeManager; attempt directories pruned by aggressive cleanup while the task finishes; a volume remounted read-only; quota exhaustion on the spill partition.","solutions":["Check the parent path in the message: confirm the NM local dir it lives under exists, is writable, and has free space","Fix ownership/permissions of yarn.nodemanager.local-dirs on that host and restart the NodeManager if needed","Exclude or repair the volume if it has gone read-only","Rerun the job once the node is healthy"],"exampleFix":"# before: single local dir, full or unwritable by the NodeManager\nyarn.nodemanager.local-dirs=/mnt/disk0/yarn-local\n\n# after: multiple writable dirs with headroom, correct ownership\nyarn.nodemanager.local-dirs=/mnt/disk0/yarn-local,/mnt/disk1/yarn-local\n# chown yarn:yarn /mnt/disk*/yarn-local ; df -h shows free space","handlingStrategy":"validation","validationCode":"// NM health script excerpt: every local dir writable with a create/delete probe\nfor (String d : conf.getStrings(\"yarn.nodemanager.local-dirs\")) {\n  File probe = new File(d, \".probe-\" + UUID.randomUUID());\n  if (!probe.createNewFile()) throw new IOException(\"local dir not writable: \" + d);\n  probe.delete();\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Enable yarn.nodemanager.disk-health-checker so failing volumes are dropped before tasks finalize","Provide multiple NM local dirs so one full disk cannot strand single-spill finalization","Keep local dirs owned by the NM user and monitor free space and inodes"],"tags":["hadoop","mapreduce","spill","rename","local-dirs","disk-full"],"backgroundTag":"file-rename-failed","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}