apache/hadoop · warning · IOException

Cannot delete block pool, it contains some block files

Error message

Cannot delete block pool, it contains some block files

What it means

FsDatasetSpi.deleteBlockPool(bpid, force=false) removes a retired block pool's directories only when they are empty: it probes volume.isBPDirEmpty(bpid) on every volume and throws if any block, meta, or file remains. It guards against silently destroying blocks the NameNode has not invalidated yet.

Solutions

1. Wait for the DataNode's async block deletions to drain (watch delete metrics / volume trash dirs), then retry the operation.
2. If the block pool is permanently retired, delete with force=true - deleteBlockPool(bpid, true) skips the emptiness check and removes the BP directories recursively.
3. Check the NN has no pending invalidation blocks for this DataNode before forcing.
4. As a last resort, stop the DN and remove the <volume>/BP-* directories manually, preserving the layout.

Example fix

// before
fsDataset.deleteBlockPool(bpid, false); // throws while blocks remain

// after
// nameservice permanently retired and invalidation drained:
fsDataset.deleteBlockPool(bpid, true); // force: recursively delete BP dirs

boolean allEmpty = true;
for (FsVolumeSpi v : fsDataset.getVolumes()) {
  allEmpty &= ((FsVolumeImpl) v).isBPDirEmpty(bpid);
}
if (!allEmpty) {
  // drain deletions first, or use force=true when the pool is permanently retired
}

try {
  fsDataset.deleteBlockPool(bpid, false);
} catch (IOException e) {
  // blocks remain: wait for invalidation to drain, then retry or force
}

When it happens

Trigger: DataNode dropping a block pool (refreshNamenodes removing a nameservice) while finalized blocks remain on volumes; block invalidation queue not drained; async deletion backlog still working through blocks.

Common situations: Decommissioning or removing a nameservice; hot-removing a NameNode from dfs.nameservices; leftover blocks after rollback.

Related errors

• Failed to delete
• Failed to delete
• block pool is not found
• cannot locate OfferService thread for bp=
• No block pool offer service for bpid=

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/31e83ed04639ed4a. Report an issue: GitHub.

Appendix: source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java:3409

      innerInfo.put("numBlocks", v.numBlocks);
      innerInfo.put("storageType", v.storageType);
      info.put(v.directory, innerInfo);
    }
    return info;
  }

  @Override //FsDatasetSpi
  public void deleteBlockPool(String bpid, boolean force)
      throws IOException {
    try (AutoCloseableLock lock = lockManager.writeLock(LockLevel.BLOCK_POOl, bpid)) {
      List<FsVolumeImpl> curVolumes = volumes.getVolumes();
      if (!force) {
        for (FsVolumeImpl volume : curVolumes) {
          try (FsVolumeReference ref = volume.obtainReference()) {
            if (!volume.isBPDirEmpty(bpid)) {
              LOG.warn(bpid
                  + " has some block files, cannot delete unless forced");
              throw new IOException("Cannot delete block pool, "
                  + "it contains some block files");
            }
          } catch (ClosedChannelException e) {
            // ignore.
          }
        }
      }
      for (FsVolumeImpl volume : curVolumes) {
        try (FsVolumeReference ref = volume.obtainReference()) {
          volume.deleteBPDirectories(bpid, force);
        } catch (ClosedChannelException e) {
          // ignore.
        }
      }
    }
  }
  
  @Override // FsDatasetSpi

View on GitHub (pinned to 2add963021)