{"record":{"id":"3beeffd8e36679cf","repo":"apache/hadoop","slug":"failed-to-delete-finalizeddir","errorCode":null,"errorMessage":"Failed to delete {finalizedDir}","messagePattern":"Failed to delete (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java","lineNumber":1183,"sourceCode":"    }\n    File tmpDir = new File(bpDir, DataStorage.STORAGE_DIR_TMP);\n    File bpCurrentDir = new File(bpDir, DataStorage.STORAGE_DIR_CURRENT);\n    File finalizedDir = new File(bpCurrentDir,\n        DataStorage.STORAGE_DIR_FINALIZED);\n    File lazypersistDir = new File(bpCurrentDir,\n        DataStorage.STORAGE_DIR_LAZY_PERSIST);\n    File rbwDir = new File(bpCurrentDir, DataStorage.STORAGE_DIR_RBW);\n    if (force) {\n      fileIoProvider.fullyDelete(this, bpDir);\n    } else {\n      if (!fileIoProvider.delete(this, rbwDir)) {\n        throw new IOException(\"Failed to delete \" + rbwDir);\n      }\n      if (!DatanodeUtil.dirNoFilesRecursive(\n              this, finalizedDir, fileIoProvider) ||\n          !fileIoProvider.fullyDelete(\n              this, finalizedDir)) {\n        throw new IOException(\"Failed to delete \" + finalizedDir);\n      }\n      if (lazypersistDir.exists() &&\n          ((!DatanodeUtil.dirNoFilesRecursive(\n              this, lazypersistDir, fileIoProvider) ||\n              !fileIoProvider.fullyDelete(this, lazypersistDir)))) {\n        throw new IOException(\"Failed to delete \" + lazypersistDir);\n      }\n      fileIoProvider.fullyDelete(this, tmpDir);\n      for (File f : fileIoProvider.listFiles(this, bpCurrentDir)) {\n        if (!fileIoProvider.delete(this, f)) {\n          throw new IOException(\"Failed to delete \" + f);\n        }\n      }\n      if (!fileIoProvider.delete(this, bpCurrentDir)) {\n        throw new IOException(\"Failed to delete \" + bpCurrentDir);\n      }\n      for (File f : fileIoProvider.listFiles(this, bpDir)) {\n        if (!fileIoProvider.delete(this, f)) {","sourceCodeStart":1165,"sourceCodeEnd":1201,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java#L1165-L1201","documentation":"Non-force block pool removal requires the finalized directory to be empty: DatanodeUtil.dirNoFilesRecursive must find no block files anywhere under finalized/, and fullyDelete must succeed. This IOException means finalized blocks remain or deletion failed - the guard that keeps deleteBlockPool(false) from destroying live data.","triggerScenarios":"Block invalidation from the NN has not drained (blocks still under finalized/subdirX); async deletion backlog lagging; files reappearing from volume trash during retry; IO errors on delete.","commonSituations":"Removing a nameservice before its blocks are invalidated; large async delete queues; read-only mounts.","solutions":["Wait for the DN's async delete queue to drain (monitor async-delete metrics and volume trash dirs), then retry.","Run the DirectoryScanner or restart the DN to reconcile volume map versus disk and release stragglers.","Confirm the NN has no pending invalidations for this DataNode and pool.","If the pool is permanently retired and the data is expendable, use force=true (deleteBlockPool(bpid, true)), which skips all emptiness checks."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"if (!DatanodeUtil.dirNoFilesRecursive(fsVolume, finalizedDir, fileIoProvider)) {\n  // finalized blocks remain: wait for invalidation / async deletes to drain\n  return;\n}","typeGuard":null,"tryCatchPattern":"try {\n  fsDataset.deleteBlockPool(bpid, false);\n} catch (IOException e) {\n  if (e.getMessage().contains(\"finalized\")) {\n    // blocks remain: wait for delete queue, then retry; force=true only if data is expendable\n  }\n}","preventionTips":["Monitor DN async delete metrics and wait for the queue to drain before pool removal.","Run the DirectoryScanner to reconcile volume map and disk before cleanup.","Force-delete only after confirming the NN no longer needs any replica in that pool."],"tags":["hdfs","datanode","block-pool","cleanup","finalized-blocks"],"backgroundTag":"block-pool-cleanup-failed","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}