apache/hadoop · error · IOException

Failed to delete {bpDir}

Error message

Failed to delete {bpDir}

What it means

Thrown by FsVolumeImpl.deleteBPDirectories() (FsVolumeImpl.java:1206) as the final step of non-force block pool removal: deleting the block pool root directory (volume/current/<bpid>) itself failed after all children were processed. Like the sibling checks it means File.delete() returned false - directory not actually empty, permissions on the directory inode, an open handle on the dir, or a read-only/failing filesystem.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java:1206

              !fileIoProvider.fullyDelete(this, lazypersistDir)))) {
        throw new IOException("Failed to delete " + lazypersistDir);
      }
      fileIoProvider.fullyDelete(this, tmpDir);
      for (File f : fileIoProvider.listFiles(this, bpCurrentDir)) {
        if (!fileIoProvider.delete(this, f)) {
          throw new IOException("Failed to delete " + f);
        }
      }
      if (!fileIoProvider.delete(this, bpCurrentDir)) {
        throw new IOException("Failed to delete " + bpCurrentDir);
      }
      for (File f : fileIoProvider.listFiles(this, bpDir)) {
        if (!fileIoProvider.delete(this, f)) {
          throw new IOException("Failed to delete " + f);
        }
      }
      if (!fileIoProvider.delete(this, bpDir)) {
        throw new IOException("Failed to delete " + bpDir);
      }
    }
  }

  @Override
  public String getStorageID() {
    return storageID;
  }

  @Override
  public StorageType getStorageType() {
    return storageType;
  }

  DatanodeStorage toDatanodeStorage() {
    return new DatanodeStorage(storageID, DatanodeStorage.State.NORMAL, storageType);
  }

View on GitHub (pinned to 2add963021)

Solutions

1. ls -la <bpDir> to see what reappeared or remains, remove/stop it, then retry deleteBlockPool
2. chown the bpDir inode to the DataNode user and retry
3. Move any shell/process out of the directory (lsof | grep <bpDir>) and retry
4. Re-run with force: hdfs dfsadmin -deleteBlockPool <dn> <bpid> force
5. Verify mount read-write status and disk health; fix or decommission the volume if the disk is failing

Example fix

# before: final bpDir rmdir fails, deleteBPDirectories aborts
hdfs dfsadmin -deleteBlockPool dn1.hadoop:9866 BP-1234567890

# after: ensure nothing recreates entries, then force delete
lsof +D /data/dfs/dn/current/BP-1234567890   # clear handles
hdfs dfsadmin -deleteBlockPool dn1.hadoop:9866 BP-1234567890 force

File bpDir = new File(volume.getCurrentDir(), bpid);
boolean deletable = bpDir.isDirectory()
    && java.nio.file.Files.isWritable(bpDir.getParentFile().toPath())
    && (bpDir.list() == null || bpDir.list().length == 0);
if (!deletable) {
  LOG.warn("bpDir {} not empty/writable; final rmdir in deleteBlockPool would fail", bpDir);
}

try {
  volume.deleteBPDirectories(bpid, false);
} catch (IOException e) {
  // last step failed: children were removed but bpDir remains - safe to rm manually or retry with force
  LOG.warn("bpDir removal failed for bp {}: {}", bpid, e);
}

When it happens

Trigger: deleteBlockPool racing with something recreating an entry directly under bpDir; another process cwd'd into bpDir (keeps rmdir busy on some filesystems); wrong ownership of the bpDir inode; NFS/NFS-gateway semantics or a remounted-read-only disk.

Common situations: Scripts or admins sitting inside the block pool directory; monitoring agents touching bpDir; data dirs restored from backup with wrong ownership; disk faults causing the FS to remount RO mid-cleanup.

Related errors

• Failed to delete {lazypersistDir}
• Failed to delete {f}
• Failed to delete {bpCurrentDir}
• Cannot remove directory {}
• Failed to rename {} to {}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/fc989b4976ffea00. Report an issue: GitHub.