apache/hadoop · warning · IOException

Cannot delete block pool, it contains some block files

Error message

Cannot delete block pool, it contains some block files

What it means

FsDatasetSpi.deleteBlockPool(bpid, force=false) removes a retired block pool's directories only when they are empty: it probes volume.isBPDirEmpty(bpid) on every volume and throws if any block, meta, or file remains. It guards against silently destroying blocks the NameNode has not invalidated yet.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java:3409

      innerInfo.put("numBlocks", v.numBlocks);
      innerInfo.put("storageType", v.storageType);
      info.put(v.directory, innerInfo);
    }
    return info;
  }

  @Override //FsDatasetSpi
  public void deleteBlockPool(String bpid, boolean force)
      throws IOException {
    try (AutoCloseableLock lock = lockManager.writeLock(LockLevel.BLOCK_POOl, bpid)) {
      List<FsVolumeImpl> curVolumes = volumes.getVolumes();
      if (!force) {
        for (FsVolumeImpl volume : curVolumes) {
          try (FsVolumeReference ref = volume.obtainReference()) {
            if (!volume.isBPDirEmpty(bpid)) {
              LOG.warn(bpid
                  + " has some block files, cannot delete unless forced");
              throw new IOException("Cannot delete block pool, "
                  + "it contains some block files");
            }
          } catch (ClosedChannelException e) {
            // ignore.
          }
        }
      }
      for (FsVolumeImpl volume : curVolumes) {
        try (FsVolumeReference ref = volume.obtainReference()) {
          volume.deleteBPDirectories(bpid, force);
        } catch (ClosedChannelException e) {
          // ignore.
        }
      }
    }
  }
  
  @Override // FsDatasetSpi

View on GitHub (pinned to 2add963021)

Solutions

1. Wait for the DataNode's async block deletions to drain (watch delete metrics / volume trash dirs), then retry the operation.
2. If the block pool is permanently retired, delete with force=true - deleteBlockPool(bpid, true) skips the emptiness check and removes the BP directories recursively.
3. Check the NN has no pending invalidation blocks for this DataNode before forcing.
4. As a last resort, stop the DN and remove the <volume>/BP-* directories manually, preserving the layout.

Example fix

// before
fsDataset.deleteBlockPool(bpid, false); // throws while blocks remain

// after
// nameservice permanently retired and invalidation drained:
fsDataset.deleteBlockPool(bpid, true); // force: recursively delete BP dirs

boolean allEmpty = true;
for (FsVolumeSpi v : fsDataset.getVolumes()) {
  allEmpty &= ((FsVolumeImpl) v).isBPDirEmpty(bpid);
}
if (!allEmpty) {
  // drain deletions first, or use force=true when the pool is permanently retired
}

try {
  fsDataset.deleteBlockPool(bpid, false);
} catch (IOException e) {
  // blocks remain: wait for invalidation to drain, then retry or force
}

When it happens

Trigger: DataNode dropping a block pool (refreshNamenodes removing a nameservice) while finalized blocks remain on volumes; block invalidation queue not drained; async deletion backlog still working through blocks.

Common situations: Decommissioning or removing a nameservice; hot-removing a NameNode from dfs.nameservices; leftover blocks after rollback.

Related errors

• Failed to delete {rbwDir}
• Failed to delete {finalizedDir}
• Storage directory for location {} and block pool id {} does
• No block pool offer service for bpid={}
• cannot locate OfferService thread for bp={blockPoolId}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/31e83ed04639ed4a. Report an issue: GitHub.