apache/hadoop · error · IOException

Failed to delete {finalizedDir}

Error message

Failed to delete {finalizedDir}

What it means

Non-force block pool removal requires the finalized directory to be empty: DatanodeUtil.dirNoFilesRecursive must find no block files anywhere under finalized/, and fullyDelete must succeed. This IOException means finalized blocks remain or deletion failed - the guard that keeps deleteBlockPool(false) from destroying live data.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java:1183

    }
    File tmpDir = new File(bpDir, DataStorage.STORAGE_DIR_TMP);
    File bpCurrentDir = new File(bpDir, DataStorage.STORAGE_DIR_CURRENT);
    File finalizedDir = new File(bpCurrentDir,
        DataStorage.STORAGE_DIR_FINALIZED);
    File lazypersistDir = new File(bpCurrentDir,
        DataStorage.STORAGE_DIR_LAZY_PERSIST);
    File rbwDir = new File(bpCurrentDir, DataStorage.STORAGE_DIR_RBW);
    if (force) {
      fileIoProvider.fullyDelete(this, bpDir);
    } else {
      if (!fileIoProvider.delete(this, rbwDir)) {
        throw new IOException("Failed to delete " + rbwDir);
      }
      if (!DatanodeUtil.dirNoFilesRecursive(
              this, finalizedDir, fileIoProvider) ||
          !fileIoProvider.fullyDelete(
              this, finalizedDir)) {
        throw new IOException("Failed to delete " + finalizedDir);
      }
      if (lazypersistDir.exists() &&
          ((!DatanodeUtil.dirNoFilesRecursive(
              this, lazypersistDir, fileIoProvider) ||
              !fileIoProvider.fullyDelete(this, lazypersistDir)))) {
        throw new IOException("Failed to delete " + lazypersistDir);
      }
      fileIoProvider.fullyDelete(this, tmpDir);
      for (File f : fileIoProvider.listFiles(this, bpCurrentDir)) {
        if (!fileIoProvider.delete(this, f)) {
          throw new IOException("Failed to delete " + f);
        }
      }
      if (!fileIoProvider.delete(this, bpCurrentDir)) {
        throw new IOException("Failed to delete " + bpCurrentDir);
      }
      for (File f : fileIoProvider.listFiles(this, bpDir)) {
        if (!fileIoProvider.delete(this, f)) {

View on GitHub (pinned to 2add963021)

Solutions

1. Wait for the DN's async delete queue to drain (monitor async-delete metrics and volume trash dirs), then retry.
2. Run the DirectoryScanner or restart the DN to reconcile volume map versus disk and release stragglers.
3. Confirm the NN has no pending invalidations for this DataNode and pool.
4. If the pool is permanently retired and the data is expendable, use force=true (deleteBlockPool(bpid, true)), which skips all emptiness checks.

if (!DatanodeUtil.dirNoFilesRecursive(fsVolume, finalizedDir, fileIoProvider)) {
  // finalized blocks remain: wait for invalidation / async deletes to drain
  return;
}

try {
  fsDataset.deleteBlockPool(bpid, false);
} catch (IOException e) {
  if (e.getMessage().contains("finalized")) {
    // blocks remain: wait for delete queue, then retry; force=true only if data is expendable
  }
}

When it happens

Trigger: Block invalidation from the NN has not drained (blocks still under finalized/subdirX); async deletion backlog lagging; files reappearing from volume trash during retry; IO errors on delete.

Common situations: Removing a nameservice before its blocks are invalidated; large async delete queues; read-only mounts.

Related errors

• Cannot delete block pool, it contains some block files
• Failed to delete {rbwDir}
• Storage directory for location {} and block pool id {} does
• No block pool offer service for bpid={}
• cannot locate OfferService thread for bp={blockPoolId}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/3beeffd8e36679cf. Report an issue: GitHub.