apache/hadoop · error · IOException

Failed to create directory {}

Error message

Failed to create directory {}

What it means

While restoring block files from the rolling-upgrade trash directory back into current/ (restoreBlockFilesFromTrash), mkdirs() for the target restore directory failed — the directory hierarchy for the restored block could not be created.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceStorage.java:569

      throws  IOException {
    int filesRestored = 0;
    File[] children = trashRoot.exists() ? trashRoot.listFiles() : null;
    if (children == null) {
      return 0;
    }

    File restoreDirectory = null;
    for (File child : children) {
      if (child.isDirectory()) {
        // Recurse to process subdirectories.
        filesRestored += restoreBlockFilesFromTrash(child);
        continue;
      }

      if (restoreDirectory == null) {
        restoreDirectory = new File(getRestoreDirectory(child));
        if (!restoreDirectory.exists() && !restoreDirectory.mkdirs()) {
          throw new IOException("Failed to create directory " + restoreDirectory);
        }
      }

      final File newChild = new File(restoreDirectory, child.getName());

      if (newChild.exists() && newChild.length() >= child.length()) {
        // Failsafe - we should not hit this case but let's make sure
        // we never overwrite a newer version of a block file with an
        // older version.
        LOG.info("Not overwriting {} with smaller file from " +
            "trash directory. This message can be safely ignored.", newChild);
      } else if (!child.renameTo(newChild)) {
        throw new IOException("Failed to rename " + child + " to " + newChild);
      } else {
        ++filesRestored;
      }
    }
    FileUtil.fullyDelete(trashRoot);

View on GitHub (pinned to 2add963021)

Solutions

1. Check free space (df) and inodes on the volume; free space if needed
2. Fix ownership/permissions so the DN user can write under current/finalized/
3. Remove any non-directory file occupying the target path, then retry the rollback

Example fix

df -h /dfs/dn
chown -R hdfs:hdfs /dfs/dn/current/BP-*/current
hdfs --daemon start datanode -rollback

// Pre-rollback checks:
if (targetDir.getUsableSpace() < minFreeBytes) throw new IOException("insufficient space");
if (!Files.isWritable(targetDir.getCanonicalFile().getParentFile().toPath())) {
  throw new IOException("no write permission for restore target");
}

catch (IOException e) {
  if (e.getMessage().contains("Failed to create directory")) {
    // stop the rollback cleanly; operator frees space/fixes perms, then retry
    abortRollbackForOperatorIntervention(e);
  }
}

When it happens

Trigger: No write permission or no space on the volume; a non-directory file exists at the path where the directory should be created; read-only mount during rollback.

Common situations: Rolling-upgrade rollback on a full disk; DN user changed since the upgrade; leftover stray files colliding with the restore path.

Related errors

• Failed to rename {} to {}
• Detached directory {} is not empty. Please manually move eac
• Cannot remove directory {}
• Cannot create directory {curDir}
• too many temporary directories.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/a9fb00f403cab8c3. Report an issue: GitHub.