apache/hadoop · error · IOException

Failed to delete {lazypersistDir}

Error message

Failed to delete {lazypersistDir}

What it means

Thrown by FsVolumeImpl.deleteBPDirectories() (FsVolumeImpl.java:1189) when a DataNode removes a block pool's directories from a volume without force mode. The lazy-persist directory (bpDir/current/lazy-persist, where LAZY_PERSIST replicas evicted from RAM_DISK are stored) either still contains files (DatanodeUtil.dirNoFilesRecursive returns false) or FileIoProvider.fullyDelete() fails to remove it. This path runs asynchronously during block pool deletion (e.g. after the NameNode is decommissioned and 'hdfs dfsadmin -deleteBlockPool' is issued without the force flag).

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java:1189

        DataStorage.STORAGE_DIR_LAZY_PERSIST);
    File rbwDir = new File(bpCurrentDir, DataStorage.STORAGE_DIR_RBW);
    if (force) {
      fileIoProvider.fullyDelete(this, bpDir);
    } else {
      if (!fileIoProvider.delete(this, rbwDir)) {
        throw new IOException("Failed to delete " + rbwDir);
      }
      if (!DatanodeUtil.dirNoFilesRecursive(
              this, finalizedDir, fileIoProvider) ||
          !fileIoProvider.fullyDelete(
              this, finalizedDir)) {
        throw new IOException("Failed to delete " + finalizedDir);
      }
      if (lazypersistDir.exists() &&
          ((!DatanodeUtil.dirNoFilesRecursive(
              this, lazypersistDir, fileIoProvider) ||
              !fileIoProvider.fullyDelete(this, lazypersistDir)))) {
        throw new IOException("Failed to delete " + lazypersistDir);
      }
      fileIoProvider.fullyDelete(this, tmpDir);
      for (File f : fileIoProvider.listFiles(this, bpCurrentDir)) {
        if (!fileIoProvider.delete(this, f)) {
          throw new IOException("Failed to delete " + f);
        }
      }
      if (!fileIoProvider.delete(this, bpCurrentDir)) {
        throw new IOException("Failed to delete " + bpCurrentDir);
      }
      for (File f : fileIoProvider.listFiles(this, bpDir)) {
        if (!fileIoProvider.delete(this, f)) {
          throw new IOException("Failed to delete " + f);
        }
      }
      if (!fileIoProvider.delete(this, bpDir)) {
        throw new IOException("Failed to delete " + bpDir);
      }

View on GitHub (pinned to 2add963021)

Solutions

1. Check directory permissions/ownership on the volume's current/lazy-persist path and chown -R hdfs:hdfs (or the DataNode user) the block pool directory, then retry deleteBlockPool
2. List the directory to see what remains (lsof +<pid> or fuser -v) and stop whatever holds files open, then retry
3. Confirm replicas for that block pool are gone first; the non-force path intentionally refuses to delete non-empty dirs
4. If leftover content is expected and unwanted, re-run with the force flag: hdfs dfsadmin -deleteBlockPool <dn> <bpid> force
5. Verify the mount is read-write (mount | grep <volume>) and the disk is healthy (dmesg for I/O errors)

Example fix

# before: fails when lazy-persist dir still holds files or perms block delete
hdfs dfsadmin -deleteBlockPool dn1.hadoop:9866 BP-1234567890

# after: fix ownership, confirm dir contents, then delete (force removes residue)
chown -R hdfs:hdfs /data/dfs/dn/current/BP-1234567890/current/lazy-persist
hdfs dfsadmin -deleteBlockPool dn1.hadoop:9866 BP-1234567890 force

// before issuing deleteBlockPool (non-force), confirm the lazy-persist dir is empty and writable
File bpDir = new File(volume.getCurrentDir(), bpid);
File lazy = new File(bpDir, "current/lazy-persist");
if (lazy.exists() && !DatanodeUtil.dirNoFilesRecursive(null, lazy, fileIoProvider)) {
  LOG.warn("lazy-persist still holds files; deleteBlockPool would fail for {}", bpid);
}
if (lazy.exists() && !(lazy.canWrite() || java.nio.file.Files.isWritable(lazy.toPath()))) {
  LOG.warn("{} not writable by this user; fix ownership first", lazy);
}

try {
  volume.deleteBPDirectories(bpid, false);
} catch (IOException e) {
  // log and keep the block pool dir for a later retry; escalate to force only after content is reviewed
  LOG.warn("Block pool cleanup failed for {} on {}: {}", bpid, volume, e);
}

When it happens

Trigger: Issuing hdfs dfsadmin -deleteBlockPool <dn-host:port> <bpid> (no force) while lazy-persist replicas still sit under current/lazy-persist; directory owned by another user so File.delete() returns false; a file handle still open on the directory (NFS or an external scanner); the disk mounted read-only or failing.

Common situations: Data directories chowned incorrectly or the DataNode runs as a different user after a rebuild; leftover lazy-persist data from an old RAM_DISK configuration; deleteBlockPool executed before all replicas for that block pool were cleaned up; storage on NFS or a RO-mounted disk at delete time.

Related errors

• Failed to delete {f}
• Failed to delete {bpCurrentDir}
• Failed to delete {bpDir}
• Cannot remove directory {}
• Failed to rename {} to {}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/344582accea085f4. Report an issue: GitHub.