apache/hadoop · error · IOException

Storage directory for location {} and block pool id {} does

Error message

Storage directory for location {} and block pool id {} does not exist

What it means

While loading block pool storage, StorageDirectory.analyzeStorage returned NON_EXISTENT: the directory for this block pool at the given location is absent and the startup option is not FORMAT, so there is nothing to recover or transition and loadStorageDirectory throws.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceStorage.java:164

   * @return
   * @throws IOException
   */
  private StorageDirectory loadStorageDirectory(NamespaceInfo nsInfo,
      StorageLocation location, StartupOption startOpt,
      List<Callable<StorageDirectory>> callables, Configuration conf)
          throws IOException {
    StorageDirectory sd = new StorageDirectory(
        nsInfo.getBlockPoolID(), null, true, location);
    try {
      StorageState curState = sd.analyzeStorage(startOpt, this, true);
      // sd is locked but not opened
      switch (curState) {
      case NORMAL:
        break;
      case NON_EXISTENT:
        LOG.info("Block pool storage directory for location {} and block pool"
            + " id {} does not exist", location, nsInfo.getBlockPoolID());
        throw new IOException("Storage directory for location " + location +
            " and block pool id " + nsInfo.getBlockPoolID() +
            " does not exist");
      case NOT_FORMATTED: // format
        LOG.info("Block pool storage directory for location {} and block pool"
                + " id {} is not formatted. Formatting ...", location,
            nsInfo.getBlockPoolID());
        format(sd, nsInfo);
        break;
      default:  // recovery part is common
        sd.doRecover(curState);
      }

      // 2. Do transitions
      // Each storage directory is treated individually.
      // During startup some of them can upgrade or roll back
      // while others could be up-to-date for the regular startup.
      if (!doTransition(sd, nsInfo, startOpt, callables, conf)) {

View on GitHub (pinned to 2add963021)

Solutions

1. Restart the DN so fresh registration formats the block pool directory for the current bpid
2. Restore the missing BP directory from backup if the blocks it held matter
3. Remove the bad entry from dfs.datanode.data.dir or correct the path

Example fix

# if the NN was re-formatted and data is disposable:
rm -rf /dfs/dn/current/BP-*   # stale block pool dirs
hdfs --daemon restart datanode  # re-registers and formats new BP dir

// Pre-flight before DN start when not formatting:
for (String dir : conf.getTrimmedStringCollection(DFS_DATANODE_DATA_DIR)) {
  File bpDir = new File(dir, "current/" + expectedBpid);
  if (!bpDir.exists() && startOpt != StartupOption.FORMAT) {
    LOG.warn("Missing block pool dir {} - DN will fail to load it", bpDir);
  }
}

When it happens

Trigger: Starting the DN with -regular/-rollback/-upgrade against a data dir where <location>/<bpid> is missing; the BP dir was deleted or moved; the NN was re-formatted so a new bpid is expected while startOpt suppresses formatting.

Common situations: Post-reformat cleanup where stale BP dirs were removed but not the reverse; operators pruning BP directories by hand; rollback attempted without a preserved previous snapshot.

Related errors

• Incompatible node types: storageType={storageType} but Stora
• Datanode CTime (={}) is not equal to namenode CTime (={})
• BlockPoolSliceStorage.recoverTransitionRead: attempt to load
• file VERSION is invalid.
• Unexpected blockpoolID {}. Expected {}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/6c0426b50bbe732b. Report an issue: GitHub.