{"record":{"id":"fc989b4976ffea00","repo":"apache/hadoop","slug":"failed-to-delete-bpdir","errorCode":null,"errorMessage":"Failed to delete {bpDir}","messagePattern":"Failed to delete (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java","lineNumber":1206,"sourceCode":"              !fileIoProvider.fullyDelete(this, lazypersistDir)))) {\n        throw new IOException(\"Failed to delete \" + lazypersistDir);\n      }\n      fileIoProvider.fullyDelete(this, tmpDir);\n      for (File f : fileIoProvider.listFiles(this, bpCurrentDir)) {\n        if (!fileIoProvider.delete(this, f)) {\n          throw new IOException(\"Failed to delete \" + f);\n        }\n      }\n      if (!fileIoProvider.delete(this, bpCurrentDir)) {\n        throw new IOException(\"Failed to delete \" + bpCurrentDir);\n      }\n      for (File f : fileIoProvider.listFiles(this, bpDir)) {\n        if (!fileIoProvider.delete(this, f)) {\n          throw new IOException(\"Failed to delete \" + f);\n        }\n      }\n      if (!fileIoProvider.delete(this, bpDir)) {\n        throw new IOException(\"Failed to delete \" + bpDir);\n      }\n    }\n  }\n\n  @Override\n  public String getStorageID() {\n    return storageID;\n  }\n\n  @Override\n  public StorageType getStorageType() {\n    return storageType;\n  }\n\n  DatanodeStorage toDatanodeStorage() {\n    return new DatanodeStorage(storageID, DatanodeStorage.State.NORMAL, storageType);\n  }\n","sourceCodeStart":1188,"sourceCodeEnd":1224,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java#L1188-L1224","documentation":"Thrown by FsVolumeImpl.deleteBPDirectories() (FsVolumeImpl.java:1206) as the final step of non-force block pool removal: deleting the block pool root directory (volume/current/<bpid>) itself failed after all children were processed. Like the sibling checks it means File.delete() returned false - directory not actually empty, permissions on the directory inode, an open handle on the dir, or a read-only/failing filesystem.","triggerScenarios":"deleteBlockPool racing with something recreating an entry directly under bpDir; another process cwd'd into bpDir (keeps rmdir busy on some filesystems); wrong ownership of the bpDir inode; NFS/NFS-gateway semantics or a remounted-read-only disk.","commonSituations":"Scripts or admins sitting inside the block pool directory; monitoring agents touching bpDir; data dirs restored from backup with wrong ownership; disk faults causing the FS to remount RO mid-cleanup.","solutions":["ls -la <bpDir> to see what reappeared or remains, remove/stop it, then retry deleteBlockPool","chown the bpDir inode to the DataNode user and retry","Move any shell/process out of the directory (lsof | grep <bpDir>) and retry","Re-run with force: hdfs dfsadmin -deleteBlockPool <dn> <bpid> force","Verify mount read-write status and disk health; fix or decommission the volume if the disk is failing"],"exampleFix":"# before: final bpDir rmdir fails, deleteBPDirectories aborts\nhdfs dfsadmin -deleteBlockPool dn1.hadoop:9866 BP-1234567890\n\n# after: ensure nothing recreates entries, then force delete\nlsof +D /data/dfs/dn/current/BP-1234567890   # clear handles\nhdfs dfsadmin -deleteBlockPool dn1.hadoop:9866 BP-1234567890 force","handlingStrategy":"validation","validationCode":"File bpDir = new File(volume.getCurrentDir(), bpid);\nboolean deletable = bpDir.isDirectory()\n    && java.nio.file.Files.isWritable(bpDir.getParentFile().toPath())\n    && (bpDir.list() == null || bpDir.list().length == 0);\nif (!deletable) {\n  LOG.warn(\"bpDir {} not empty/writable; final rmdir in deleteBlockPool would fail\", bpDir);\n}","typeGuard":null,"tryCatchPattern":"try {\n  volume.deleteBPDirectories(bpid, false);\n} catch (IOException e) {\n  // last step failed: children were removed but bpDir remains - safe to rm manually or retry with force\n  LOG.warn(\"bpDir removal failed for bp {}: {}\", bpid, e);\n}","preventionTips":["Make sure no shell/process has cwd inside the block pool dir during deletion","Prefer the force flag when the remaining content is known garbage","Keep ownership of the whole data-dir tree consistent (chown -R on provisioning)"],"tags":["hdfs","datanode","block-pool-removal","filesystem"],"backgroundTag":"file-delete-failed","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}