{"record":{"id":"9163968b99a3d42f","repo":"apache/hadoop","slug":"error-formatting-datanode-dirs","errorCode":null,"errorMessage":"Error formatting DataNode dirs","messagePattern":"Error formatting DataNode dirs","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-infra/src/main/java/org/apache/hadoop/tools/dynamometer/SimulatedDataNodes.java","lineNumber":126,"sourceCode":"      printUsageExit(\"No NameNode address and port in config\");\n    }\n    System.out.println(\"DataNodes will connect to NameNode at \" + nameNodeAdr);\n\n    String loc = DataNode.getStorageLocations(getConf()).get(0).toString();\n    loc = loc.substring(loc.indexOf(\"]\") + 1); // delete storage type\n    String path = new URI(loc).getPath();\n    System.setProperty(MiniDFSCluster.PROP_TEST_BUILD_DATA, path);\n    SimulatedFSDataset.setFactory(getConf());\n    getConf().setLong(SimulatedFSDataset.CONFIG_PROPERTY_CAPACITY,\n        STORAGE_CAPACITY);\n\n    UserGroupInformation.setConfiguration(getConf());\n    MiniDFSCluster mc = new MiniDFSCluster();\n    try {\n      mc.formatDataNodeDirs();\n    } catch (IOException e) {\n      System.out.println(\"Error formatting DataNode dirs: \" + e);\n      throw new RuntimeException(\"Error formatting DataNode dirs\", e);\n    }\n\n    try {\n      System.out.println(\"Found \" + blockListFiles.size()\n          + \" block listing files; launching DataNodes accordingly.\");\n      mc.startDataNodes(getConf(), blockListFiles.size(), null, false,\n          StartupOption.REGULAR, null, null, null, null, false, true, true,\n          null, null, null);\n      long startTime = Time.monotonicNow();\n      System.out.println(\"Waiting for DataNodes to connect to NameNode and \"\n          + \"init storage directories.\");\n      Set<DataNode> datanodesWithoutFSDataset = new HashSet<>(\n          mc.getDataNodes());\n      while (!datanodesWithoutFSDataset.isEmpty()) {\n        datanodesWithoutFSDataset\n            .removeIf((dn) -> DataNodeTestUtils.getFSDataset(dn) != null);\n        Thread.sleep(100);\n      }","sourceCodeStart":108,"sourceCodeEnd":144,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-infra/src/main/java/org/apache/hadoop/tools/dynamometer/SimulatedDataNodes.java#L108-L144","documentation":"SimulatedDataNodes derives a base path from the block list location, sets it as MiniDFSCluster's test.build.data property, and calls MiniDFSCluster.formatDataNodeDirs() to wipe/prepare the DataNode storage directories. Any IOException from that formatting is caught, printed, and rewrapped in this RuntimeException.","triggerScenarios":"The storage directory under the derived test.build.data path cannot be formatted: no write permission, leftover dfs/data storage from a previous run holding an incompatible VERSION/clusterID, a file where a directory is expected, or no space on the volume.","commonSituations":"Re-running SimulatedDataNodes without cleaning the prior run's <path>/dfs/data, running as a user without write access to the block-list parent directory, or a read-only mount.","solutions":["Remove leftover storage: rm -rf <blockListParent>/dfs/data (the path set as test.build.data)","chmod/chown the parent path so the running user can create and write directories","Confirm the location URI in the argument resolves to an existing absolute directory","Check free space on the volume hosting the path"],"exampleFix":"# before: rerun with stale MiniDFSCluster storage left in place\nhadoop ... SimulatedDataNodes bp-1 /data/dn0/blocks.txt\n# after: clean the derived storage dir first\nrm -rf /data/dn0/dfs/data\nhadoop ... SimulatedDataNodes bp-1 /data/dn0/blocks.txt","handlingStrategy":"validation","validationCode":"Path base = new URI(blockListLocation).getPath();\nPath storage = new Path(base, \"dfs/data\");\nif (storage.getFileSystem(conf).exists(storage)) {\n  storage.getFileSystem(conf).delete(storage, true); // clean stale VERSION/clusterID\n}\nif (!java.nio.file.Files.isWritable(new java.io.File(base).toPath())) {\n  throw new IllegalStateException(\"not writable: \" + base);\n}","typeGuard":null,"tryCatchPattern":"catch (RuntimeException e) { /* unwrap IOException cause; after fixing perms/cleaning storage, a fresh process run is required */ }","preventionTips":["Always wipe the derived <blockListParent>/dfs/data between SimulatedDataNodes runs","Run as the same user that owns the block-list files","Keep the block-list location on a volume with free space and write access"],"tags":["dynamometer","minidfscluster","storage","permissions"],"backgroundTag":"storage-directory-unwritable","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}