apache/hadoop · error · IOException

Failed to delete {f}

Error message

Failed to delete {f}

What it means

Thrown by FsVolumeImpl.deleteBPDirectories() (FsVolumeImpl.java:1194) while iterating direct children of the block pool's current/ directory (e.g. VERSION, finalized, rbw already handled, scanner cursor files). FileIoProvider.delete() returned false, meaning java.io.File.delete() refused the file: permission denied, an open file handle, or a failing filesystem. The exception aborts the rest of the non-force block pool cleanup.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java:1194

      if (!fileIoProvider.delete(this, rbwDir)) {
        throw new IOException("Failed to delete " + rbwDir);
      }
      if (!DatanodeUtil.dirNoFilesRecursive(
              this, finalizedDir, fileIoProvider) ||
          !fileIoProvider.fullyDelete(
              this, finalizedDir)) {
        throw new IOException("Failed to delete " + finalizedDir);
      }
      if (lazypersistDir.exists() &&
          ((!DatanodeUtil.dirNoFilesRecursive(
              this, lazypersistDir, fileIoProvider) ||
              !fileIoProvider.fullyDelete(this, lazypersistDir)))) {
        throw new IOException("Failed to delete " + lazypersistDir);
      }
      fileIoProvider.fullyDelete(this, tmpDir);
      for (File f : fileIoProvider.listFiles(this, bpCurrentDir)) {
        if (!fileIoProvider.delete(this, f)) {
          throw new IOException("Failed to delete " + f);
        }
      }
      if (!fileIoProvider.delete(this, bpCurrentDir)) {
        throw new IOException("Failed to delete " + bpCurrentDir);
      }
      for (File f : fileIoProvider.listFiles(this, bpDir)) {
        if (!fileIoProvider.delete(this, f)) {
          throw new IOException("Failed to delete " + f);
        }
      }
      if (!fileIoProvider.delete(this, bpDir)) {
        throw new IOException("Failed to delete " + bpDir);
      }
    }
  }

  @Override
  public String getStorageID() {

View on GitHub (pinned to 2add963021)

Solutions

1. Fix ownership/permissions on bpDir/current (chown -R <dn-user>:<dn-group>), then retry deleteBlockPool
2. Find and stop processes holding the listed file open (lsof | grep <file>), then retry
3. Retry once after a few seconds to rule out a file recreated between list and delete
4. Use the force flag to fully delete remaining content: hdfs dfsadmin -deleteBlockPool <dn> <bpid> force
5. Check dmesg/mount status for read-only remount or I/O errors on that volume

Example fix

# before: File.delete() fails on current/ child, cleanup aborts
hdfs dfsadmin -deleteBlockPool dn1.hadoop:9866 BP-1234567890

# after: release handles, fix perms, then force-delete remainder
lsof +D /data/dfs/dn/current/BP-1234567890/current   # stop offenders
chown -R hdfs:hdfs /data/dfs/dn/current/BP-1234567890
hdfs dfsadmin -deleteBlockPool dn1.hadoop:9866 BP-1234567890 force

File current = new File(bpDir, "current");
for (File f : current.listFiles()) {
  if (!f.canWrite() && !java.nio.file.Files.isWritable(f.toPath())) {
    LOG.warn("{} not writable; deleteBlockPool would abort on it", f);
  }
}

try {
  volume.deleteBPDirectories(bpid, false);
} catch (IOException e) {
  LOG.warn("Aborted deleting child of {} for bp {}: {}", currentDir, bpid, e);
  // retry once after a short delay to rule out files recreated mid-loop; then consider force
}

When it happens

Trigger: hdfs dfsadmin -deleteBlockPool without force while current/ still contains a file the DataNode user cannot delete; a concurrently recreated file (dir quota/AV scan) between listFiles() and delete(); NFS silly-rename leaving the unlink pending; read-only mount.

Common situations: Wrong ownership of data dirs after re-provisioning; security agents (antivirus, audit) holding files open; deleteBlockPool racing with the DataNode still writing a final VERSION/scanner file; disk media errors making unlink fail.

Related errors

• Failed to delete {lazypersistDir}
• Failed to delete {bpCurrentDir}
• Failed to delete {bpDir}
• Cannot remove directory {}
• Failed to rename {} to {}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/a9360dd7dd155a87. Report an issue: GitHub.