{"record":{"id":"248c32da4fc2c872","repo":"apache/hadoop","slug":"failed-to-delete-rbwdir","errorCode":null,"errorMessage":"Failed to delete {rbwDir}","messagePattern":"Failed to delete (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java","lineNumber":1177,"sourceCode":"  void deleteBPDirectories(String bpid, boolean force) throws IOException {\n    File volumeCurrentDir = this.getCurrentDir();\n    File bpDir = new File(volumeCurrentDir, bpid);\n    if (!bpDir.isDirectory()) {\n      // nothing to be deleted\n      return;\n    }\n    File tmpDir = new File(bpDir, DataStorage.STORAGE_DIR_TMP);\n    File bpCurrentDir = new File(bpDir, DataStorage.STORAGE_DIR_CURRENT);\n    File finalizedDir = new File(bpCurrentDir,\n        DataStorage.STORAGE_DIR_FINALIZED);\n    File lazypersistDir = new File(bpCurrentDir,\n        DataStorage.STORAGE_DIR_LAZY_PERSIST);\n    File rbwDir = new File(bpCurrentDir, DataStorage.STORAGE_DIR_RBW);\n    if (force) {\n      fileIoProvider.fullyDelete(this, bpDir);\n    } else {\n      if (!fileIoProvider.delete(this, rbwDir)) {\n        throw new IOException(\"Failed to delete \" + rbwDir);\n      }\n      if (!DatanodeUtil.dirNoFilesRecursive(\n              this, finalizedDir, fileIoProvider) ||\n          !fileIoProvider.fullyDelete(\n              this, finalizedDir)) {\n        throw new IOException(\"Failed to delete \" + finalizedDir);\n      }\n      if (lazypersistDir.exists() &&\n          ((!DatanodeUtil.dirNoFilesRecursive(\n              this, lazypersistDir, fileIoProvider) ||\n              !fileIoProvider.fullyDelete(this, lazypersistDir)))) {\n        throw new IOException(\"Failed to delete \" + lazypersistDir);\n      }\n      fileIoProvider.fullyDelete(this, tmpDir);\n      for (File f : fileIoProvider.listFiles(this, bpCurrentDir)) {\n        if (!fileIoProvider.delete(this, f)) {\n          throw new IOException(\"Failed to delete \" + f);\n        }","sourceCodeStart":1159,"sourceCodeEnd":1195,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java#L1159-L1195","documentation":"FsVolumeImpl.deleteBPDirectories(bpid, force=false) removes the block pool's rbw directory first; this IOException means the RBW directory could not be deleted - typically because replica files being written (or orphaned by crashed writers) still sit in it, or an IO error (permissions, read-only mount) blocked removal.","triggerScenarios":"Deleting a block pool while RBW replicas exist (writes in flight or orphaned from crashed clients); volume read-only or out of inodes; another process holding files open.","commonSituations":"Removing a nameservice while jobs still write through it; unclean writer shutdowns leaving rbw residue; NFS-mounted volumes.","solutions":["Drain or stop writers to that block pool, then retry the removal.","If the pool is definitely retired, stop the DN and delete leftover rbw contents under <volume>/BP-*/current/rbw manually.","Check the volume mount is writable and has free inodes and space.","Or remove the whole block pool with force=true, which deletes the entire BP directory tree unconditionally."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"File rbwDir = new File(bpDir, \"current/\" + DataStorage.STORAGE_DIR_RBW);\nif (rbwDir.exists() && rbwDir.list().length > 0) {\n  // live or orphaned RBW replicas: stop writers / clean up before non-force delete\n}","typeGuard":null,"tryCatchPattern":"try {\n  fsDataset.deleteBlockPool(bpid, false);\n} catch (IOException e) {\n  if (e.getMessage().contains(\"rbw\")) {\n    // drain writers, then retry; or force=true if the pool is retired\n  }\n}","preventionTips":["Stop or drain writers to a block pool before retiring it.","Confirm the volume is writable and has free inodes before deletion.","Use force=true only when the BP directory tree may be destroyed wholesale."],"tags":["hdfs","datanode","block-pool","cleanup","rbw"],"backgroundTag":"block-pool-cleanup-failed","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}