apache/hadoop · error · IOException

Cannot remove directory {}

Error message

Cannot remove directory {}

What it means

In the same pre-APPEND_RBW_DIR rollback path, once the detach directory is empty it is removed with detachDir.delete(). A false return - permissions, a file appearing concurrently, or a filesystem error - triggers 'Cannot remove directory <dir>' and aborts the rollback.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java:898

   * 
   * If the directory is not empty report an error; 
   * Otherwise remove the directory.
   * 
   * @param detachDir detach directory
   * @throws IOException if the directory is not empty or it can not be removed
   */
  private void cleanupDetachDir(File detachDir) throws IOException {
    if (!DataNodeLayoutVersion.supports(
        LayoutVersion.Feature.APPEND_RBW_DIR, layoutVersion) &&
        detachDir.exists() && detachDir.isDirectory() ) {
      
        if (FileUtil.list(detachDir).length != 0 ) {
          throw new IOException("Detached directory " + detachDir +
              " is not empty. Please manually move each file under this " +
              "directory to the finalized directory if the finalized " +
              "directory tree does not have the file.");
        } else if (!detachDir.delete()) {
          throw new IOException("Cannot remove directory " + detachDir);
        }
    }
  }
  
  /** 
   * Rolling back to a snapshot in previous directory by moving it to current
   * directory.
   * Rollback procedure:
   * <br>
   * If previous directory exists:
   * <ol>
   * <li> Rename current to removed.tmp </li>
   * <li> Rename previous to current </li>
   * <li> Remove removed.tmp </li>
   * </ol>
   * 
   * If previous directory does not exist and the current version supports
   * federation, perform a simple rollback of layout version. This does not

View on GitHub (pinned to 2add963021)

Solutions

1. Fix ownership/permissions so the DN user can modify the parent directory, then retry rollback
2. Remove the empty detach directory manually as the DN user and retry
3. Do not use NFS for DN storage; check mount health (mount, dmesg) on that host

// before rollback: DN user must be able to modify the parent of detach/
Path detach = dir.resolve("detach");
if (Files.exists(detach)) {
  Path parent = detach.getParent();
  if (!Files.isWritable(parent)) {
    throw new IOException("Cannot modify " + parent + " - fix ownership/permissions");
  }
}

catch (IOException e) {
  if (String.valueOf(e.getMessage()).contains("Cannot remove directory")) {
    // permissions/mount issue: fix perms on the parent dir or remove detach/ manually, then retry
  } else { throw e; }
}

When it happens

Trigger: Rollback with an empty but undeletable detach directory: no write permission on the parent directory for the DN user, NFS/mount transient errors, or another process touching detach/ mid-rollback.

Common situations: Wrong ownership of the data dir tree; NFS-mounted DN storage with hiccups; racing processes creating files inside detach/.

Related errors

• Cannot remove directory {}
• Failed to rename {} to {}
• Storage directory with location {} does not exist
• Detached directory {} is not empty. Please manually move eac
• Permission denied: user=%s, path="%s":%s:%s:%s%s

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/f5c4b6fd3b72f730. Report an issue: GitHub.