apache/hadoop · error · IOException

BlockPoolSliceStorage.recoverTransitionRead: attempt to load

Error message

BlockPoolSliceStorage.recoverTransitionRead: attempt to load an used block storage: {}

What it means

BlockPoolSliceStorage.loadBpStorageDirectories detected via containsStorageDir that this StorageLocation is already loaded for the same block pool — i.e. one physical block-pool directory would be loaded twice.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceStorage.java:222

   * Therefore, a failure on loading any block pool storage results a faulty
   * data volume.
   *
   * @param nsInfo namespace information
   * @param location storage directories of block pool
   * @param startOpt startup option
   * @param callables list of callable storage directory
   * @param conf configuration
   * @return an array of loaded block pool directories.
   * @throws IOException on error
   */
  List<StorageDirectory> loadBpStorageDirectories(NamespaceInfo nsInfo,
      StorageLocation location, StartupOption startOpt,
      List<Callable<StorageDirectory>> callables, Configuration conf)
          throws IOException {
    List<StorageDirectory> succeedDirs = Lists.newArrayList();
    try {
      if (containsStorageDir(location, nsInfo.getBlockPoolID())) {
        throw new IOException(
            "BlockPoolSliceStorage.recoverTransitionRead: " +
                "attempt to load an used block storage: " + location);
      }
      final StorageDirectory sd = loadStorageDirectory(
          nsInfo, location, startOpt, callables, conf);
      succeedDirs.add(sd);
    } catch (IOException e) {
      LOG.warn("Failed to analyze storage directories for block pool {}",
          nsInfo.getBlockPoolID(), e);
      throw e;
    }
    return succeedDirs;
  }

  /**
   * Analyze storage directories. Recover from previous transitions if required.
   *
   * The block pool storages are either all analyzed or none of them is loaded.

View on GitHub (pinned to 2add963021)

Solutions

Deduplicate dfs.datanode.data.dir so every entry is a distinct real directory (resolve symlinks)
Give each entry its own disk/mount point
Restart the DN after fixing the configuration

Example fix

<!-- before -->
<property><name>dfs.datanode.data.dir</name>
  <value>/dfs/dn,/dfs/dn,/dfs/dn2</value></property>

<!-- after -->
<property><name>dfs.datanode.data.dir</name>
  <value>/dfs/dn,/dfs/dn2</value></property>

Defensive patterns

Strategy: validation

Validate before calling

Set<Path> seen = new HashSet<>();
for (String dir : conf.getTrimmedStringCollection(DFS_DATANODE_DATA_DIR)) {
  Path canon = new File(dir).getCanonicalFile().toPath();
  if (!seen.add(canon)) {
    throw new IllegalArgumentException("Duplicate data dir (after symlink resolution): " + dir);
  }
}

Prevention

Lint dfs.datanode.data.dir for duplicates and symlink aliases before deploying
Generate data.dir from a disk inventory (one entry per real device) rather than by hand

When it happens

Trigger: The same path listed multiple times in dfs.datanode.data.dir; entries that differ textually (symlink, trailing slash, relative vs absolute, bind mount) but resolve to the same directory.

Common situations: Copy-paste duplication in data.dir; /data and /mnt/data symlinked; templating bugs that emit one volume per disk plus a duplicate.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/4ce6240d44915a9f. Report an issue: GitHub.