{"record":{"id":"52bad725fa98111d","repo":"apache/hadoop","slug":"failed-to-delete-bpcurrentdir","errorCode":null,"errorMessage":"Failed to delete {bpCurrentDir}","messagePattern":"Failed to delete (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java","lineNumber":1198,"sourceCode":"              this, finalizedDir, fileIoProvider) ||\n          !fileIoProvider.fullyDelete(\n              this, finalizedDir)) {\n        throw new IOException(\"Failed to delete \" + finalizedDir);\n      }\n      if (lazypersistDir.exists() &&\n          ((!DatanodeUtil.dirNoFilesRecursive(\n              this, lazypersistDir, fileIoProvider) ||\n              !fileIoProvider.fullyDelete(this, lazypersistDir)))) {\n        throw new IOException(\"Failed to delete \" + lazypersistDir);\n      }\n      fileIoProvider.fullyDelete(this, tmpDir);\n      for (File f : fileIoProvider.listFiles(this, bpCurrentDir)) {\n        if (!fileIoProvider.delete(this, f)) {\n          throw new IOException(\"Failed to delete \" + f);\n        }\n      }\n      if (!fileIoProvider.delete(this, bpCurrentDir)) {\n        throw new IOException(\"Failed to delete \" + bpCurrentDir);\n      }\n      for (File f : fileIoProvider.listFiles(this, bpDir)) {\n        if (!fileIoProvider.delete(this, f)) {\n          throw new IOException(\"Failed to delete \" + f);\n        }\n      }\n      if (!fileIoProvider.delete(this, bpDir)) {\n        throw new IOException(\"Failed to delete \" + bpDir);\n      }\n    }\n  }\n\n  @Override\n  public String getStorageID() {\n    return storageID;\n  }\n\n  @Override","sourceCodeStart":1180,"sourceCodeEnd":1216,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java#L1180-L1216","documentation":"Thrown by FsVolumeImpl.deleteBPDirectories() (FsVolumeImpl.java:1198) when FileIoProvider.delete() fails on the block pool's current/ directory itself after every listed child was supposedly removed. It means the directory was not empty at unlink time (a file or subdir appeared between the loop and this call, or listFiles() hid an undeletable entry) or the filesystem refused the rmdir (permissions, read-only, NFS).","triggerScenarios":"deleteBlockPool (non-force) racing with a thread recreating a file inside current/; a hidden/subdirectory entry the child loop could not remove (its own 'Failed to delete {f}' would normally fire first); NFS or FUSE semantics where rmdir of a busy dir returns EACCES/EBUSY.","commonSituations":"Delete issued while the DataNode is still finalizing writes for that block pool; data dirs on NFS with silly-rename; wrong ownership of the current/ directory inode itself; disk remounted read-only after errors.","solutions":["Confirm nothing recreated files under bpDir/current (ls -la), stop the writer, then retry deleteBlockPool","Fix ownership of the directory itself, not just children: chown <dn-user> <bpDir>/current, then retry","Check the mount is read-write and healthy (mount, dmesg | grep -i error)","Use force mode if the residue is known: hdfs dfsadmin -deleteBlockPool <dn> <bpid> force","As a last resort stop the DataNode, rm -rf the block pool dir manually, restart"],"exampleFix":"# before: current/ dir itself undeletable, IOException aborts cleanup\nhdfs dfsadmin -deleteBlockPool dn1.hadoop:9866 BP-1234567890\n\n# after: verify empty, fix dir ownership, force delete\nls -la /data/dfs/dn/current/BP-1234567890/current\nchown hdfs:hdfs /data/dfs/dn/current/BP-1234567890/current\nhdfs dfsadmin -deleteBlockPool dn1.hadoop:9866 BP-1234567890 force","handlingStrategy":"validation","validationCode":"File current = new File(bpDir, \"current\");\n// re-check emptiness immediately before the delete call, not minutes before\nboolean empty = current.isDirectory()\n    && current.list() != null && current.list().length == 0;\nif (!empty) {\n  LOG.warn(\"current/ not empty at delete time for bp {}\", bpid);\n}","typeGuard":null,"tryCatchPattern":"try {\n  volume.deleteBPDirectories(bpid, false);\n} catch (IOException e) {\n  if (e.getMessage().contains(\"current\")) {\n    // list what reappeared, stop the writer, then retry; else force-delete\n    LOG.warn(\"current/ delete failed: {}\", e);\n  }\n}","preventionTips":["Quiesce writers for the block pool before issuing deleteBlockPool","Keep directory inodes owned by the DataNode user, not just their contents","Avoid NFS-backed data dirs where rmdir of busy dirs fails"],"tags":["hdfs","datanode","block-pool-removal","filesystem"],"backgroundTag":"file-delete-failed","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}