apache/hadoop · critical · IOException

All specified directories have failed to load.

Error message

All specified directories have failed to load.

What it means

recoverTransitionRead is the per-block-pool startup path: it attempts every configured dfs.datanode.data.dir via addStorageLocations and requires at least one success. If the returned list is empty - every single directory failed for its own individually-logged reason - this aggregate IOException is thrown and datanode startup aborts.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java:567

  /**
   * Analyze storage directories for a specific block pool.
   * Recover from previous transitions if required.
   * Perform fs state transition if necessary depending on the namespace info.
   * Read storage info.
   * <br>
   * This method should be synchronized between multiple DN threads.  Only the
   * first DN thread does DN level storage dir recoverTransitionRead.
   *
   * @param datanode DataNode
   * @param nsInfo Namespace info of namenode corresponding to the block pool
   * @param dataDirs Storage directories
   * @param startOpt startup option
   * @throws IOException on error
   */
  void recoverTransitionRead(DataNode datanode, NamespaceInfo nsInfo,
      Collection<StorageLocation> dataDirs, StartupOption startOpt) throws IOException {
    if (addStorageLocations(datanode, nsInfo, dataDirs, startOpt).isEmpty()) {
      throw new IOException("All specified directories have failed to load.");
    }
  }

  void format(StorageDirectory sd, NamespaceInfo nsInfo,
              String newDatanodeUuid, Configuration conf) throws IOException {
    sd.clearDirectory(); // create directory
    this.layoutVersion = DataNodeLayoutVersion.getCurrentLayoutVersion();
    this.clusterID = nsInfo.getClusterID();
    this.namespaceID = nsInfo.getNamespaceID();
    this.cTime = 0;
    setDatanodeUuid(newDatanodeUuid);

    createStorageID(sd, false, conf);
    writeProperties(sd);
  }

  /*
   * Set ClusterID, StorageID, StorageType, CTime into

View on GitHub (pinned to 2add963021)

Solutions

1. Read the log lines immediately above this exception - each directory's specific failure is logged separately and is the real cause; fix those
2. Verify recursive ownership/permissions: chown -R hdfs:hdfs <dir>; chmod 700 <dir>
3. If the NN was re-formatted intentionally, deliberately clear/reformat the DN storage (destroys that node's replica metadata) and restart
4. If a mount is simply missing, remount and restart - do not wipe data unnecessarily

// Pre-start smoke test for every configured data dir
for (String dir : conf.getTrimmedStrings(DFS_DATANODE_DATA_DIR_KEY)) {
  Path p = Paths.get(dir);
  if (!Files.isDirectory(p)) throw new IOException("Missing: " + p);
  if (!Files.isReadable(p) || !Files.isWritable(p)) throw new IOException("Bad perms: " + p);
  Path version = p.resolve("current/VERSION");
  if (Files.exists(version) && Files.readAllLines(version).stream()
        .noneMatch(l -> l.startsWith("clusterID"))) throw new IOException("No clusterID in " + version);
}

catch (IOException e) {
  if ("All specified directories have failed to load.".equals(e.getMessage())) {
    // aggregate failure: scan preceding log lines for each dir's root cause
  } else { throw e; }
}

When it happens

Trigger: Starting a datanode where 100% of configured storage dirs fail: nonexistent or permission-denied paths, incompatible clusterID, corrupt VERSION files, locked directories, or hardware I/O errors on all disks simultaneously.

Common situations: Single-data-dir node whose disk died or was wiped; clusterID mismatch after NN re-format on a one-dir DN; wrong ownership of data dirs after user/host migration; all dirs sitting on an unmounted device.

Related errors

• Failed to remove %s: %s
• Root {}: DatanodeUuid={}, does not match {} from other Stora
• Incompatible node types: storageType={storageType} but Stora
• Cluster IDs not matched: dn cid={clusterId} but ns cid={nsCi
• Although a UNIX domain socket path is configured as {domainS

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/5fb25a3580a3bbe2. Report an issue: GitHub.