{"record":{"id":"5fb25a3580a3bbe2","repo":"apache/hadoop","slug":"all-specified-directories-have-failed-to-load","errorCode":null,"errorMessage":"All specified directories have failed to load.","messagePattern":"All specified directories have failed to load\\.","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"critical","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java","lineNumber":567,"sourceCode":"  /**\n   * Analyze storage directories for a specific block pool.\n   * Recover from previous transitions if required.\n   * Perform fs state transition if necessary depending on the namespace info.\n   * Read storage info.\n   * <br>\n   * This method should be synchronized between multiple DN threads.  Only the\n   * first DN thread does DN level storage dir recoverTransitionRead.\n   *\n   * @param datanode DataNode\n   * @param nsInfo Namespace info of namenode corresponding to the block pool\n   * @param dataDirs Storage directories\n   * @param startOpt startup option\n   * @throws IOException on error\n   */\n  void recoverTransitionRead(DataNode datanode, NamespaceInfo nsInfo,\n      Collection<StorageLocation> dataDirs, StartupOption startOpt) throws IOException {\n    if (addStorageLocations(datanode, nsInfo, dataDirs, startOpt).isEmpty()) {\n      throw new IOException(\"All specified directories have failed to load.\");\n    }\n  }\n\n  void format(StorageDirectory sd, NamespaceInfo nsInfo,\n              String newDatanodeUuid, Configuration conf) throws IOException {\n    sd.clearDirectory(); // create directory\n    this.layoutVersion = DataNodeLayoutVersion.getCurrentLayoutVersion();\n    this.clusterID = nsInfo.getClusterID();\n    this.namespaceID = nsInfo.getNamespaceID();\n    this.cTime = 0;\n    setDatanodeUuid(newDatanodeUuid);\n\n    createStorageID(sd, false, conf);\n    writeProperties(sd);\n  }\n\n  /*\n   * Set ClusterID, StorageID, StorageType, CTime into","sourceCodeStart":549,"sourceCodeEnd":585,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java#L549-L585","documentation":"recoverTransitionRead is the per-block-pool startup path: it attempts every configured dfs.datanode.data.dir via addStorageLocations and requires at least one success. If the returned list is empty - every single directory failed for its own individually-logged reason - this aggregate IOException is thrown and datanode startup aborts.","triggerScenarios":"Starting a datanode where 100% of configured storage dirs fail: nonexistent or permission-denied paths, incompatible clusterID, corrupt VERSION files, locked directories, or hardware I/O errors on all disks simultaneously.","commonSituations":"Single-data-dir node whose disk died or was wiped; clusterID mismatch after NN re-format on a one-dir DN; wrong ownership of data dirs after user/host migration; all dirs sitting on an unmounted device.","solutions":["Read the log lines immediately above this exception - each directory's specific failure is logged separately and is the real cause; fix those","Verify recursive ownership/permissions: chown -R hdfs:hdfs <dir>; chmod 700 <dir>","If the NN was re-formatted intentionally, deliberately clear/reformat the DN storage (destroys that node's replica metadata) and restart","If a mount is simply missing, remount and restart - do not wipe data unnecessarily"],"exampleFix":null,"handlingStrategy":"validation","validationCode":"// Pre-start smoke test for every configured data dir\nfor (String dir : conf.getTrimmedStrings(DFS_DATANODE_DATA_DIR_KEY)) {\n  Path p = Paths.get(dir);\n  if (!Files.isDirectory(p)) throw new IOException(\"Missing: \" + p);\n  if (!Files.isReadable(p) || !Files.isWritable(p)) throw new IOException(\"Bad perms: \" + p);\n  Path version = p.resolve(\"current/VERSION\");\n  if (Files.exists(version) && Files.readAllLines(version).stream()\n        .noneMatch(l -> l.startsWith(\"clusterID\"))) throw new IOException(\"No clusterID in \" + version);\n}","typeGuard":null,"tryCatchPattern":"catch (IOException e) {\n  if (\"All specified directories have failed to load.\".equals(e.getMessage())) {\n    // aggregate failure: scan preceding log lines for each dir's root cause\n  } else { throw e; }\n}","preventionTips":["Never re-format only the NameNode in shared/test clusters; clear DN dirs in the same procedure","Keep DN data dirs owned by the DN user with mode 700, verified by config management","Monitor the count of successfully loaded volumes at startup so partial failures surface before total failure"],"tags":["hadoop","hdfs","datanode","storage","startup","fatal"],"backgroundTag":"storage-initialization-failed","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}