apache/hadoop · error · IOException

Detached directory {} is not empty. Please manually move eac

Error message

Detached directory {} is not empty. Please manually move each file under this directory to the finalized directory if the finalized directory tree does not have the file.

What it means

During rollback of layouts older than APPEND_RBW_DIR, cleanupDetachDir inspects the legacy detach directory. If it still contains files (leftovers from the old append/copy-on-write mechanics), the DN cannot tell whether those blocks also exist in finalized, so it refuses the rollback and asks the operator to reconcile the files manually.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java:893

    writeProperties(sd);
  }

  /**
   * Cleanup the detachDir. 
   * 
   * If the directory is not empty report an error; 
   * Otherwise remove the directory.
   * 
   * @param detachDir detach directory
   * @throws IOException if the directory is not empty or it can not be removed
   */
  private void cleanupDetachDir(File detachDir) throws IOException {
    if (!DataNodeLayoutVersion.supports(
        LayoutVersion.Feature.APPEND_RBW_DIR, layoutVersion) &&
        detachDir.exists() && detachDir.isDirectory() ) {
      
        if (FileUtil.list(detachDir).length != 0 ) {
          throw new IOException("Detached directory " + detachDir +
              " is not empty. Please manually move each file under this " +
              "directory to the finalized directory if the finalized " +
              "directory tree does not have the file.");
        } else if (!detachDir.delete()) {
          throw new IOException("Cannot remove directory " + detachDir);
        }
    }
  }
  
  /** 
   * Rolling back to a snapshot in previous directory by moving it to current
   * directory.
   * Rollback procedure:
   * <br>
   * If previous directory exists:
   * <ol>
   * <li> Rename current to removed.tmp </li>
   * <li> Rename previous to current </li>

View on GitHub (pinned to 2add963021)

Solutions

Follow the message: for each file under detach/, move it into the finalized tree only if finalized does not already contain that block
Once detach/ is empty, retry the rollback
Back up detach/ before doing any manual moves

Defensive patterns

Strategy: validation

Validate before calling

// before 'hdfs datanode -rollback' on legacy layouts
Path detach = dir.resolve("detach");
if (Files.isDirectory(detach)
    && Files.list(detach).findAny().isPresent()) {
  throw new IOException("detach/ not empty under " + dir
      + " - reconcile files into finalized/ first (skip blocks finalized already has)");
}

Prevention

Before rollback rehearsals on old clusters, audit every data dir for leftover detach/ contents
Move detach files into finalized only when finalized lacks that block, then empty detach/
Snapshot/backup detach/ before manual reconciliation

When it happens

Trigger: Running 'hdfs datanode -rollback' on a storage dir whose layout predates APPEND_RBW_DIR while <dir>/detach still holds files from earlier append operations.

Common situations: Old clusters exercising rollback after an upgrade that had used appends; upgrade rehearsals on historical dirs; restored legacy directories containing detach leftovers.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/879818ce1a8376b8. Report an issue: GitHub.