apache/hadoop · error · IOException

Cannot create directory {}

Error message

Cannot create directory {}

What it means

IOException thrown in DataStorage's block-directory linking code (used during upgrades, e.g. moving/linking blocks into a new directory layout) when File.mkdirs() returns false for a target subdirectory that is not itself a block directory. The upgrade wants to mirror the source directory structure under the destination volume, and the OS refuses to create the directory. mkdirs() returning false (rather than throwing) usually means a path component conflicts with an existing file, a permission denial, or a read-only/full filesystem.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java:1350

      return;
    }
    // from is a directory
    hl.linkStats.countDirs++;
    
    String[] blockNames = from.list(new java.io.FilenameFilter() {
      @Override
      public boolean accept(File dir, String name) {
        return name.startsWith(Block.BLOCK_FILE_PREFIX);
      }
    });

    // If we are upgrading to block ID-based layout, we don't want to recreate
    // any subdirs from the source that contain blocks, since we have a new
    // directory structure
    if (!upgradeToIdBasedLayout || !to.getName().startsWith(
        BLOCK_SUBDIR_PREFIX)) {
      if (!to.mkdirs())
        throw new IOException("Cannot create directory " + to);
    }

    // Block files just need hard links with the same file names
    // but a different directory
    if (blockNames.length > 0) {
      if (upgradeToIdBasedLayout) {
        for (String blockName : blockNames) {
          long blockId = Block.getBlockId(blockName);
          File blockLocation = DatanodeUtil.idToBlockDir(blockRoot, blockId);
          if (!blockLocation.exists()) {
            if (!blockLocation.mkdirs()) {
              throw new IOException("Failed to mkdirs " + blockLocation);
            }
          }
          /**
           * The destination path is 32x32, so 1024 distinct paths. Therefore
           * we cache the destination path and reuse the same File object on
           * potentially thousands of blocks located on this volume.

View on GitHub (pinned to 2add963021)

Solutions

1. Inspect the destination path from the DataNode log for the exact 'to' directory and check what blocks it: ls -ld on each path component, remove a conflicting regular file if one exists
2. Fix ownership/permissions so the datanode user can create directories on that volume (chown -R hdfs:hadoop <volume>; chmod u+w)
3. Confirm the volume is mounted read-write and has free space and inodes (df -h, df -i)
4. Free space / inodes on the affected volume and restart the DataNode to retry the upgrade

Example fix

# before: upgrade fails, log shows: Cannot create directory <vol>/current/bp-.../subdir0
ls -ld <vol>/current/bp-*/subdir0   # check for a stale FILE where a dir is expected

# after: clear the conflict and permissions, restart DN
sudo rm -f <vol>/current/bp-*/subdir0   # only if it is a stale leftover file
sudo chown -R hdfs:hadoop <vol> && sudo chmod u+rwx <vol>
hdfs --daemon start datanode

// Pre-upgrade check: every volume dir must be writable by the DN user
for (File vol : new File[] { new File("/data/dfs") }) {
  File probe = new File(vol, ".upgrade-probe-" + System.currentTimeMillis());
  if (!probe.mkdir() && !probe.exists()) {
    throw new IOException("Volume not writable, upgrade will fail: " + vol);
  }
  Files.deleteIfExists(probe.toPath());
}

try {
  dataStorage.upgrade(...);  // triggers linkBlocks/moveDirBlocks
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().startsWith("Cannot create directory")) {
    // read the 'to' path from the message, check perms/space/conflicting file,
    // fix and restart DN to retry the upgrade - do not force-continue
  }
  throw e;
}

When it happens

Trigger: DataNode layout upgrade (e.g. upgrade to block-ID-based layout) calling the recursive link/move routine with a destination 'to' whose parent path cannot be created: File.mkdirs() returns false. Typical concrete causes: an existing regular file occupies a path component of 'to', no write/execute permission on the volume directory, volume mounted read-only, or inode/disk exhaustion.

Common situations: Starting a cluster upgrade on volumes with restrictive ownership (datanode user cannot write), leftover files from a failed earlier upgrade blocking directory creation, NFS/SAN volumes mounted read-only during upgrade, disk full or fs inode exhaustion.

Related errors

• Mkdirs failed to create {}
• Mkdirs failed to create ${dir}
• Cannot remove directory {}
• Cannot remove directory {}
• Failed to mkdirs {}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/20e57798a2beea8e. Report an issue: GitHub.