{"record":{"id":"31e83ed04639ed4a","repo":"apache/hadoop","slug":"cannot-delete-block-pool-it-contains-some-block-f","errorCode":null,"errorMessage":"Cannot delete block pool, it contains some block files","messagePattern":"Cannot delete block pool, it contains some block files","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"warning","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java","lineNumber":3409,"sourceCode":"      innerInfo.put(\"numBlocks\", v.numBlocks);\n      innerInfo.put(\"storageType\", v.storageType);\n      info.put(v.directory, innerInfo);\n    }\n    return info;\n  }\n\n  @Override //FsDatasetSpi\n  public void deleteBlockPool(String bpid, boolean force)\n      throws IOException {\n    try (AutoCloseableLock lock = lockManager.writeLock(LockLevel.BLOCK_POOl, bpid)) {\n      List<FsVolumeImpl> curVolumes = volumes.getVolumes();\n      if (!force) {\n        for (FsVolumeImpl volume : curVolumes) {\n          try (FsVolumeReference ref = volume.obtainReference()) {\n            if (!volume.isBPDirEmpty(bpid)) {\n              LOG.warn(bpid\n                  + \" has some block files, cannot delete unless forced\");\n              throw new IOException(\"Cannot delete block pool, \"\n                  + \"it contains some block files\");\n            }\n          } catch (ClosedChannelException e) {\n            // ignore.\n          }\n        }\n      }\n      for (FsVolumeImpl volume : curVolumes) {\n        try (FsVolumeReference ref = volume.obtainReference()) {\n          volume.deleteBPDirectories(bpid, force);\n        } catch (ClosedChannelException e) {\n          // ignore.\n        }\n      }\n    }\n  }\n  \n  @Override // FsDatasetSpi","sourceCodeStart":3391,"sourceCodeEnd":3427,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java#L3391-L3427","documentation":"FsDatasetSpi.deleteBlockPool(bpid, force=false) removes a retired block pool's directories only when they are empty: it probes volume.isBPDirEmpty(bpid) on every volume and throws if any block, meta, or file remains. It guards against silently destroying blocks the NameNode has not invalidated yet.","triggerScenarios":"DataNode dropping a block pool (refreshNamenodes removing a nameservice) while finalized blocks remain on volumes; block invalidation queue not drained; async deletion backlog still working through blocks.","commonSituations":"Decommissioning or removing a nameservice; hot-removing a NameNode from dfs.nameservices; leftover blocks after rollback.","solutions":["Wait for the DataNode's async block deletions to drain (watch delete metrics / volume trash dirs), then retry the operation.","If the block pool is permanently retired, delete with force=true - deleteBlockPool(bpid, true) skips the emptiness check and removes the BP directories recursively.","Check the NN has no pending invalidation blocks for this DataNode before forcing.","As a last resort, stop the DN and remove the <volume>/BP-* directories manually, preserving the layout."],"exampleFix":"// before\nfsDataset.deleteBlockPool(bpid, false); // throws while blocks remain\n\n// after\n// nameservice permanently retired and invalidation drained:\nfsDataset.deleteBlockPool(bpid, true); // force: recursively delete BP dirs","handlingStrategy":"validation","validationCode":"boolean allEmpty = true;\nfor (FsVolumeSpi v : fsDataset.getVolumes()) {\n  allEmpty &= ((FsVolumeImpl) v).isBPDirEmpty(bpid);\n}\nif (!allEmpty) {\n  // drain deletions first, or use force=true when the pool is permanently retired\n}","typeGuard":null,"tryCatchPattern":"try {\n  fsDataset.deleteBlockPool(bpid, false);\n} catch (IOException e) {\n  // blocks remain: wait for invalidation to drain, then retry or force\n}","preventionTips":["Drain the DN async delete backlog before retiring a block pool.","Use force=true only after the nameservice is permanently removed and its data is expendable.","Confirm the NN has no pending invalidations for the DN before forcing deletion."],"tags":["hdfs","datanode","block-pool","cleanup","refreshnamenodes"],"backgroundTag":"block-pool-cleanup-failed","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}