apache/hadoop · error · IOException

Detached directory {} is not empty. Please manually move eac

Error message

Detached directory {} is not empty. Please manually move each file under this directory to the finalized directory if the finalized directory tree does not have the file.

What it means

Legacy upgrade path: on layout versions older than APPEND_RBW_DIR, the upgrade requires the old detached/ directory (left by pre-0.21 append hard-link 'detach') to be empty. It still contains files, so cleanupDetachDir aborts the upgrade rather than silently dropping block files.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceStorage.java:536

    LOG.info("Upgrade of {} is complete", name);
  }

  /**
   * Cleanup the detachDir.
   * 
   * If the directory is not empty report an error; Otherwise remove the
   * directory.
   * 
   * @param detachDir detach directory
   * @throws IOException if the directory is not empty or it can not be removed
   */
  private void cleanupDetachDir(File detachDir) throws IOException {
    if (!DataNodeLayoutVersion.supports(
        LayoutVersion.Feature.APPEND_RBW_DIR, layoutVersion)
        && detachDir.exists() && detachDir.isDirectory()) {

      if (FileUtil.list(detachDir).length != 0) {
        throw new IOException("Detached directory " + detachDir
            + " is not empty. Please manually move each file under this "
            + "directory to the finalized directory if the finalized "
            + "directory tree does not have the file.");
      } else if (!detachDir.delete()) {
        throw new IOException("Cannot remove directory " + detachDir);
      }
    }
  }

  /**
   * Restore all files from the trash directory to their corresponding
   * locations under current/
   */
  private int restoreBlockFilesFromTrash(File trashRoot)
      throws  IOException {
    int filesRestored = 0;
    File[] children = trashRoot.exists() ? trashRoot.listFiles() : null;
    if (children == null) {

View on GitHub (pinned to 2add963021)

Solutions

1. Move each file from detached/ into the finalized/ tree only where finalized does not already have it (exactly as the message instructs)
2. Delete detached files that already exist with the same or larger size under finalized/
3. Re-run the DN upgrade once detached/ is empty

Example fix

# for f in /dfs/dn/current/BP-*/detached/*; do
#   tgt=/dfs/dn/current/BP-*/current/finalized/...  # matching subpath
#   [ -e "$tgt" ] || mv "$f" "$tgt"
# done
hdfs --daemon start datanode -upgrade

// Pre-upgrade check on legacy layouts:
File detach = new File(bpCurrentDir, "detached");
if (detach.isDirectory() && FileUtil.list(detach).length > 0) {
  throw new IOException("detached/ not empty; reconcile with finalized/ before upgrade");
}

When it happens

Trigger: Upgrading DataNode storage from a very old release that used the detach mechanism, with files still sitting in <bp>/detached/.

Common situations: Clusters upgraded across many releases in one hop; storage archives restored from ancient snapshots.

Related errors

• Detached directory {} is not empty. Please manually move eac
• version file in current directory is missing.
• too many temporary directories.
• previous and finalized.tmpcannot exist together.
• previous and previous.tmp cannot exist together.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/d53621f5d4e43f0a. Report an issue: GitHub.