{"record":{"id":"e72941110ddcb3dd","repo":"apache/hadoop","slug":"failed-to-mkdirs","errorCode":null,"errorMessage":"Failed to mkdirs {}","messagePattern":"Failed to mkdirs (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java","lineNumber":1362,"sourceCode":"    // If we are upgrading to block ID-based layout, we don't want to recreate\n    // any subdirs from the source that contain blocks, since we have a new\n    // directory structure\n    if (!upgradeToIdBasedLayout || !to.getName().startsWith(\n        BLOCK_SUBDIR_PREFIX)) {\n      if (!to.mkdirs())\n        throw new IOException(\"Cannot create directory \" + to);\n    }\n\n    // Block files just need hard links with the same file names\n    // but a different directory\n    if (blockNames.length > 0) {\n      if (upgradeToIdBasedLayout) {\n        for (String blockName : blockNames) {\n          long blockId = Block.getBlockId(blockName);\n          File blockLocation = DatanodeUtil.idToBlockDir(blockRoot, blockId);\n          if (!blockLocation.exists()) {\n            if (!blockLocation.mkdirs()) {\n              throw new IOException(\"Failed to mkdirs \" + blockLocation);\n            }\n          }\n          /**\n           * The destination path is 32x32, so 1024 distinct paths. Therefore\n           * we cache the destination path and reuse the same File object on\n           * potentially thousands of blocks located on this volume.\n           * This method is called recursively so the cache is passed through\n           * each recursive call. There is one cache per volume, and it is only\n           * accessed by a single thread so no locking is needed.\n           */\n          File cachedDest = pathCache\n              .computeIfAbsent(blockLocation, k -> blockLocation);\n          idBasedLayoutSingleLinks.add(new LinkArgs(from,\n              cachedDest, blockName));\n          hl.linkStats.countSingleLinks++;\n        }\n      } else {\n        HardLink.createHardLinkMult(from, blockNames, to);","sourceCodeStart":1344,"sourceCodeEnd":1380,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java#L1344-L1380","documentation":"IOException thrown during the same DataNode upgrade block-linking pass as 2281, but specifically when computing the new block-ID-based directory for a block file: DatanodeUtil.idToBlockDir(blockRoot, blockId) maps a block id to one of the 32x32 subdirectories, and mkdirs() on that target subdirectory fails. The upgrade cannot place the hard link for the block because its destination directory cannot be created.","triggerScenarios":"Upgrade to block-ID-based layout (upgradeToIdBasedLayout == true) with block files present: for each blockName, blockLocation = DatanodeUtil.idToBlockDir(blockRoot, blockId); if it does not exist, blockLocation.mkdirs() returns false. Causes: permission/ownership problems on blockRoot subtree, a stale file occupying the subdirectory path, read-only or full/inode-exhausted volume.","commonSituations":"First upgrade of old (pre-block-ID layout) DataNode directories on volumes with wrong ownership; leftover junk files in current/ named like subdirNN; volumes on network storage that went read-only; disk or inode exhaustion during upgrade.","solutions":["From the DataNode log take the exact 'Failed to mkdirs <dir>' path and verify each component: remove stale files blocking it, ensure it is a directory","Grant the datanode user write access to the block pool subtree (chown/chmod) and confirm the mount is read-write","Check df -h / df -i on the volume for space or inode exhaustion and clean up","Restart the DataNode to resume the upgrade once the blocking condition is removed"],"exampleFix":"# before: 'Failed to mkdirs /data/dfs/current/BP-.../subdir17' during upgrade\nls -ld /data/dfs/current/BP-*/subdir17\n\n# after: fix blocker and retry upgrade\nsudo chown -R hdfs:hadoop /data/dfs\nsudo chmod -R u+rwX /data/dfs\nrm -f /data/dfs/current/BP-*/subdir17  # only if it is a stale FILE, not a directory\nhdfs --daemon start datanode","handlingStrategy":"try-catch","validationCode":"// Verify the block-pool subtree is fully writable before upgrading old layouts\nPath bpRoot = Paths.get(volumeDir.toString(), \"current\", blockPoolId);\nfor (int i = 0; i < 32; i++) {\n  Path sub = bpRoot.resolve(\"subdir\" + i);\n  if (Files.exists(sub) && !Files.isDirectory(sub)) {\n    throw new IOException(\"Stale file blocks upgrade target: \" + sub);\n  }\n  if (!Files.isWritable(bpRoot)) {\n    throw new IOException(\"Block pool root not writable: \" + bpRoot);\n  }\n}","typeGuard":null,"tryCatchPattern":"try {\n  // DataNode startup with -upgrade linking blocks into ID-based layout\n} catch (IOException e) {\n  if (e.getMessage() != null && e.getMessage().startsWith(\"Failed to mkdirs\")) {\n    String dir = e.getMessage().substring(\"Failed to mkdirs \".length()).trim();\n    // inspect dir: remove stale file, fix ownership/mount, free inodes, then restart DN\n  }\n}","preventionTips":["Pre-upgrade audit of old-layout directories for stray files named subdirNN or anything shadowing target paths","Ensure datanode user owns the entire storage directory tree before starting an upgrade","Check df -h and df -i (inodes, not just bytes) on every volume before upgrade - hard-link layout needs many directories","Never mount data volumes read-only (or let the kernel flip them) during upgrade windows"],"tags":["hdfs","datanode","upgrade","block-id-layout","mkdirs","hardlink"],"backgroundTag":"directory-create-failed","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}