apache/hadoop · error · IOException

Failed to delete {rbwDir}

Error message

Failed to delete {rbwDir}

What it means

FsVolumeImpl.deleteBPDirectories(bpid, force=false) removes the block pool's rbw directory first; this IOException means the RBW directory could not be deleted - typically because replica files being written (or orphaned by crashed writers) still sit in it, or an IO error (permissions, read-only mount) blocked removal.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java:1177

  void deleteBPDirectories(String bpid, boolean force) throws IOException {
    File volumeCurrentDir = this.getCurrentDir();
    File bpDir = new File(volumeCurrentDir, bpid);
    if (!bpDir.isDirectory()) {
      // nothing to be deleted
      return;
    }
    File tmpDir = new File(bpDir, DataStorage.STORAGE_DIR_TMP);
    File bpCurrentDir = new File(bpDir, DataStorage.STORAGE_DIR_CURRENT);
    File finalizedDir = new File(bpCurrentDir,
        DataStorage.STORAGE_DIR_FINALIZED);
    File lazypersistDir = new File(bpCurrentDir,
        DataStorage.STORAGE_DIR_LAZY_PERSIST);
    File rbwDir = new File(bpCurrentDir, DataStorage.STORAGE_DIR_RBW);
    if (force) {
      fileIoProvider.fullyDelete(this, bpDir);
    } else {
      if (!fileIoProvider.delete(this, rbwDir)) {
        throw new IOException("Failed to delete " + rbwDir);
      }
      if (!DatanodeUtil.dirNoFilesRecursive(
              this, finalizedDir, fileIoProvider) ||
          !fileIoProvider.fullyDelete(
              this, finalizedDir)) {
        throw new IOException("Failed to delete " + finalizedDir);
      }
      if (lazypersistDir.exists() &&
          ((!DatanodeUtil.dirNoFilesRecursive(
              this, lazypersistDir, fileIoProvider) ||
              !fileIoProvider.fullyDelete(this, lazypersistDir)))) {
        throw new IOException("Failed to delete " + lazypersistDir);
      }
      fileIoProvider.fullyDelete(this, tmpDir);
      for (File f : fileIoProvider.listFiles(this, bpCurrentDir)) {
        if (!fileIoProvider.delete(this, f)) {
          throw new IOException("Failed to delete " + f);
        }

View on GitHub (pinned to 2add963021)

Solutions

1. Drain or stop writers to that block pool, then retry the removal.
2. If the pool is definitely retired, stop the DN and delete leftover rbw contents under <volume>/BP-*/current/rbw manually.
3. Check the volume mount is writable and has free inodes and space.
4. Or remove the whole block pool with force=true, which deletes the entire BP directory tree unconditionally.

File rbwDir = new File(bpDir, "current/" + DataStorage.STORAGE_DIR_RBW);
if (rbwDir.exists() && rbwDir.list().length > 0) {
  // live or orphaned RBW replicas: stop writers / clean up before non-force delete
}

try {
  fsDataset.deleteBlockPool(bpid, false);
} catch (IOException e) {
  if (e.getMessage().contains("rbw")) {
    // drain writers, then retry; or force=true if the pool is retired
  }
}

When it happens

Trigger: Deleting a block pool while RBW replicas exist (writes in flight or orphaned from crashed clients); volume read-only or out of inodes; another process holding files open.

Common situations: Removing a nameservice while jobs still write through it; unclean writer shutdowns leaving rbw residue; NFS-mounted volumes.

Related errors

• Cannot delete block pool, it contains some block files
• Failed to delete {finalizedDir}
• Storage directory for location {} and block pool id {} does
• No block pool offer service for bpid={}
• Replica is not readable, block={block}, replica={replica}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/248c32da4fc2c872. Report an issue: GitHub.