{"record":{"id":"a9360dd7dd155a87","repo":"apache/hadoop","slug":"failed-to-delete-f","errorCode":null,"errorMessage":"Failed to delete {f}","messagePattern":"Failed to delete (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java","lineNumber":1194,"sourceCode":"      if (!fileIoProvider.delete(this, rbwDir)) {\n        throw new IOException(\"Failed to delete \" + rbwDir);\n      }\n      if (!DatanodeUtil.dirNoFilesRecursive(\n              this, finalizedDir, fileIoProvider) ||\n          !fileIoProvider.fullyDelete(\n              this, finalizedDir)) {\n        throw new IOException(\"Failed to delete \" + finalizedDir);\n      }\n      if (lazypersistDir.exists() &&\n          ((!DatanodeUtil.dirNoFilesRecursive(\n              this, lazypersistDir, fileIoProvider) ||\n              !fileIoProvider.fullyDelete(this, lazypersistDir)))) {\n        throw new IOException(\"Failed to delete \" + lazypersistDir);\n      }\n      fileIoProvider.fullyDelete(this, tmpDir);\n      for (File f : fileIoProvider.listFiles(this, bpCurrentDir)) {\n        if (!fileIoProvider.delete(this, f)) {\n          throw new IOException(\"Failed to delete \" + f);\n        }\n      }\n      if (!fileIoProvider.delete(this, bpCurrentDir)) {\n        throw new IOException(\"Failed to delete \" + bpCurrentDir);\n      }\n      for (File f : fileIoProvider.listFiles(this, bpDir)) {\n        if (!fileIoProvider.delete(this, f)) {\n          throw new IOException(\"Failed to delete \" + f);\n        }\n      }\n      if (!fileIoProvider.delete(this, bpDir)) {\n        throw new IOException(\"Failed to delete \" + bpDir);\n      }\n    }\n  }\n\n  @Override\n  public String getStorageID() {","sourceCodeStart":1176,"sourceCodeEnd":1212,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java#L1176-L1212","documentation":"Thrown by FsVolumeImpl.deleteBPDirectories() (FsVolumeImpl.java:1194) while iterating direct children of the block pool's current/ directory (e.g. VERSION, finalized, rbw already handled, scanner cursor files). FileIoProvider.delete() returned false, meaning java.io.File.delete() refused the file: permission denied, an open file handle, or a failing filesystem. The exception aborts the rest of the non-force block pool cleanup.","triggerScenarios":"hdfs dfsadmin -deleteBlockPool without force while current/ still contains a file the DataNode user cannot delete; a concurrently recreated file (dir quota/AV scan) between listFiles() and delete(); NFS silly-rename leaving the unlink pending; read-only mount.","commonSituations":"Wrong ownership of data dirs after re-provisioning; security agents (antivirus, audit) holding files open; deleteBlockPool racing with the DataNode still writing a final VERSION/scanner file; disk media errors making unlink fail.","solutions":["Fix ownership/permissions on bpDir/current (chown -R <dn-user>:<dn-group>), then retry deleteBlockPool","Find and stop processes holding the listed file open (lsof | grep <file>), then retry","Retry once after a few seconds to rule out a file recreated between list and delete","Use the force flag to fully delete remaining content: hdfs dfsadmin -deleteBlockPool <dn> <bpid> force","Check dmesg/mount status for read-only remount or I/O errors on that volume"],"exampleFix":"# before: File.delete() fails on current/ child, cleanup aborts\nhdfs dfsadmin -deleteBlockPool dn1.hadoop:9866 BP-1234567890\n\n# after: release handles, fix perms, then force-delete remainder\nlsof +D /data/dfs/dn/current/BP-1234567890/current   # stop offenders\nchown -R hdfs:hdfs /data/dfs/dn/current/BP-1234567890\nhdfs dfsadmin -deleteBlockPool dn1.hadoop:9866 BP-1234567890 force","handlingStrategy":"validation","validationCode":"File current = new File(bpDir, \"current\");\nfor (File f : current.listFiles()) {\n  if (!f.canWrite() && !java.nio.file.Files.isWritable(f.toPath())) {\n    LOG.warn(\"{} not writable; deleteBlockPool would abort on it\", f);\n  }\n}","typeGuard":null,"tryCatchPattern":"try {\n  volume.deleteBPDirectories(bpid, false);\n} catch (IOException e) {\n  LOG.warn(\"Aborted deleting child of {} for bp {}: {}\", currentDir, bpid, e);\n  // retry once after a short delay to rule out files recreated mid-loop; then consider force\n}","preventionTips":["Ensure the DataNode user owns every file under the block pool directory","Issue deleteBlockPool when the DataNode is quiescent for that block pool","Audit (auditd/AV) exceptions for data dirs so they do not hold files open"],"tags":["hdfs","datanode","block-pool-removal","filesystem"],"backgroundTag":"file-delete-failed","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}