apache/hadoop · error · RuntimeException

Error formatting DataNode dirs

Error message

Error formatting DataNode dirs

What it means

SimulatedDataNodes derives a base path from the block list location, sets it as MiniDFSCluster's test.build.data property, and calls MiniDFSCluster.formatDataNodeDirs() to wipe/prepare the DataNode storage directories. Any IOException from that formatting is caught, printed, and rewrapped in this RuntimeException.

Source

Thrown at hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-infra/src/main/java/org/apache/hadoop/tools/dynamometer/SimulatedDataNodes.java:126

      printUsageExit("No NameNode address and port in config");
    }
    System.out.println("DataNodes will connect to NameNode at " + nameNodeAdr);

    String loc = DataNode.getStorageLocations(getConf()).get(0).toString();
    loc = loc.substring(loc.indexOf("]") + 1); // delete storage type
    String path = new URI(loc).getPath();
    System.setProperty(MiniDFSCluster.PROP_TEST_BUILD_DATA, path);
    SimulatedFSDataset.setFactory(getConf());
    getConf().setLong(SimulatedFSDataset.CONFIG_PROPERTY_CAPACITY,
        STORAGE_CAPACITY);

    UserGroupInformation.setConfiguration(getConf());
    MiniDFSCluster mc = new MiniDFSCluster();
    try {
      mc.formatDataNodeDirs();
    } catch (IOException e) {
      System.out.println("Error formatting DataNode dirs: " + e);
      throw new RuntimeException("Error formatting DataNode dirs", e);
    }

    try {
      System.out.println("Found " + blockListFiles.size()
          + " block listing files; launching DataNodes accordingly.");
      mc.startDataNodes(getConf(), blockListFiles.size(), null, false,
          StartupOption.REGULAR, null, null, null, null, false, true, true,
          null, null, null);
      long startTime = Time.monotonicNow();
      System.out.println("Waiting for DataNodes to connect to NameNode and "
          + "init storage directories.");
      Set<DataNode> datanodesWithoutFSDataset = new HashSet<>(
          mc.getDataNodes());
      while (!datanodesWithoutFSDataset.isEmpty()) {
        datanodesWithoutFSDataset
            .removeIf((dn) -> DataNodeTestUtils.getFSDataset(dn) != null);
        Thread.sleep(100);
      }

View on GitHub (pinned to 2add963021)

Solutions

1. Remove leftover storage: rm -rf <blockListParent>/dfs/data (the path set as test.build.data)
2. chmod/chown the parent path so the running user can create and write directories
3. Confirm the location URI in the argument resolves to an existing absolute directory
4. Check free space on the volume hosting the path

Example fix

# before: rerun with stale MiniDFSCluster storage left in place
hadoop ... SimulatedDataNodes bp-1 /data/dn0/blocks.txt
# after: clean the derived storage dir first
rm -rf /data/dn0/dfs/data
hadoop ... SimulatedDataNodes bp-1 /data/dn0/blocks.txt

Path base = new URI(blockListLocation).getPath();
Path storage = new Path(base, "dfs/data");
if (storage.getFileSystem(conf).exists(storage)) {
  storage.getFileSystem(conf).delete(storage, true); // clean stale VERSION/clusterID
}
if (!java.nio.file.Files.isWritable(new java.io.File(base).toPath())) {
  throw new IllegalStateException("not writable: " + base);
}

catch (RuntimeException e) { /* unwrap IOException cause; after fixing perms/cleaning storage, a fresh process run is required */ }

When it happens

Trigger: The storage directory under the derived test.build.data path cannot be formatted: no write permission, leftover dfs/data storage from a previous run holding an incompatible VERSION/clusterID, a file where a directory is expected, or no space on the volume.

Common situations: Re-running SimulatedDataNodes without cleaning the prior run's <path>/dfs/data, running as a user without write access to the block-list parent directory, or a read-only mount.

Related errors

• Cannot remove current directory: {curDir}
• Cannot create directory {curDir}
• Cannot create directory {rootPath}
• Failed to delete {dir}
• Cannot remove directory {}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/9163968b99a3d42f. Report an issue: GitHub.