apache/hadoop · error · IOException

Failed to mkdirs {}

Error message

Failed to mkdirs {}

What it means

IOException thrown during the same DataNode upgrade block-linking pass as 2281, but specifically when computing the new block-ID-based directory for a block file: DatanodeUtil.idToBlockDir(blockRoot, blockId) maps a block id to one of the 32x32 subdirectories, and mkdirs() on that target subdirectory fails. The upgrade cannot place the hard link for the block because its destination directory cannot be created.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java:1362

    // If we are upgrading to block ID-based layout, we don't want to recreate
    // any subdirs from the source that contain blocks, since we have a new
    // directory structure
    if (!upgradeToIdBasedLayout || !to.getName().startsWith(
        BLOCK_SUBDIR_PREFIX)) {
      if (!to.mkdirs())
        throw new IOException("Cannot create directory " + to);
    }

    // Block files just need hard links with the same file names
    // but a different directory
    if (blockNames.length > 0) {
      if (upgradeToIdBasedLayout) {
        for (String blockName : blockNames) {
          long blockId = Block.getBlockId(blockName);
          File blockLocation = DatanodeUtil.idToBlockDir(blockRoot, blockId);
          if (!blockLocation.exists()) {
            if (!blockLocation.mkdirs()) {
              throw new IOException("Failed to mkdirs " + blockLocation);
            }
          }
          /**
           * The destination path is 32x32, so 1024 distinct paths. Therefore
           * we cache the destination path and reuse the same File object on
           * potentially thousands of blocks located on this volume.
           * This method is called recursively so the cache is passed through
           * each recursive call. There is one cache per volume, and it is only
           * accessed by a single thread so no locking is needed.
           */
          File cachedDest = pathCache
              .computeIfAbsent(blockLocation, k -> blockLocation);
          idBasedLayoutSingleLinks.add(new LinkArgs(from,
              cachedDest, blockName));
          hl.linkStats.countSingleLinks++;
        }
      } else {
        HardLink.createHardLinkMult(from, blockNames, to);

View on GitHub (pinned to 2add963021)

Solutions

1. From the DataNode log take the exact 'Failed to mkdirs <dir>' path and verify each component: remove stale files blocking it, ensure it is a directory
2. Grant the datanode user write access to the block pool subtree (chown/chmod) and confirm the mount is read-write
3. Check df -h / df -i on the volume for space or inode exhaustion and clean up
4. Restart the DataNode to resume the upgrade once the blocking condition is removed

Example fix

# before: 'Failed to mkdirs /data/dfs/current/BP-.../subdir17' during upgrade
ls -ld /data/dfs/current/BP-*/subdir17

# after: fix blocker and retry upgrade
sudo chown -R hdfs:hadoop /data/dfs
sudo chmod -R u+rwX /data/dfs
rm -f /data/dfs/current/BP-*/subdir17  # only if it is a stale FILE, not a directory
hdfs --daemon start datanode

// Verify the block-pool subtree is fully writable before upgrading old layouts
Path bpRoot = Paths.get(volumeDir.toString(), "current", blockPoolId);
for (int i = 0; i < 32; i++) {
  Path sub = bpRoot.resolve("subdir" + i);
  if (Files.exists(sub) && !Files.isDirectory(sub)) {
    throw new IOException("Stale file blocks upgrade target: " + sub);
  }
  if (!Files.isWritable(bpRoot)) {
    throw new IOException("Block pool root not writable: " + bpRoot);
  }
}

try {
  // DataNode startup with -upgrade linking blocks into ID-based layout
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().startsWith("Failed to mkdirs")) {
    String dir = e.getMessage().substring("Failed to mkdirs ".length()).trim();
    // inspect dir: remove stale file, fix ownership/mount, free inodes, then restart DN
  }
}

When it happens

Trigger: Upgrade to block-ID-based layout (upgradeToIdBasedLayout == true) with block files present: for each blockName, blockLocation = DatanodeUtil.idToBlockDir(blockRoot, blockId); if it does not exist, blockLocation.mkdirs() returns false. Causes: permission/ownership problems on blockRoot subtree, a stale file occupying the subdirectory path, read-only or full/inode-exhausted volume.

Common situations: First upgrade of old (pre-block-ID layout) DataNode directories on volumes with wrong ownership; leftover junk files in current/ named like subdirNN; volumes on network storage that went read-only; disk or inode exhaustion during upgrade.

Related errors

• Cannot create directory {}
• Detached directory {} is not empty. Please manually move eac
• DATANODE_STATUS_NOT_REGULAR
• dfs.datanode.parallel.volumes.load.threads.num = {} < 1
• Cannot rollback to a newer state. Datanode previous state: L

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/e72941110ddcb3dd. Report an issue: GitHub.