apache/hadoop · error · IOException

Storage directory with location {} does not exist

Error message

Storage directory with location {} does not exist

What it means

During DataStorage.loadStorageDirectory, StorageDirectory.analyzeStorage reports NON_EXISTENT when the configured location does not exist as a directory and is not being created on this path (non-FORMAT startup). The DN logs the location and aborts loading that directory with this IOException.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java:281

        }
        storage.addStorageDir(sd);
      }
    }
  }

  private StorageDirectory loadStorageDirectory(DataNode datanode,
      NamespaceInfo nsInfo, StorageLocation location, StartupOption startOpt,
      List<Callable<StorageDirectory>> callables) throws IOException {
    StorageDirectory sd = new StorageDirectory(null, false, location);
    try {
      StorageState curState = sd.analyzeStorage(startOpt, this, true);
      // sd is locked but not opened
      switch (curState) {
      case NORMAL:
        break;
      case NON_EXISTENT:
        LOG.info("Storage directory with location {} does not exist", location);
        throw new IOException("Storage directory with location " + location
            + " does not exist");
      case NOT_FORMATTED: // format
        LOG.info("Storage directory with location {} is not formatted for "
            + "namespace {}. Formatting...", location, nsInfo.getNamespaceID());
        format(sd, nsInfo, datanode.getDatanodeUuid(), datanode.getConf());
        break;
      default:  // recovery part is common
        sd.doRecover(curState);
      }

      // 2. Do transitions
      // Each storage directory is treated individually.
      // During startup some of them can upgrade or roll back
      // while others could be up-to-date for the regular startup.
      if (!doTransition(sd, nsInfo, startOpt, callables, datanode.getConf())) {

        // 3. Update successfully loaded storage.
        setServiceLayoutVersion(getServiceLayoutVersion());

View on GitHub (pinned to 2add963021)

Solutions

1. Verify the path exists and the DN user can create/write it: sudo -u hdfs mkdir -p <dir> && sudo -u hdfs touch <dir>/.write_test
2. Fix typos in dfs.datanode.data.dir
3. Restore the missing mount (fstab / k8s volume) and restart the DN
4. If the disk is dead, remove that entry from dfs.datanode.data.dir before restarting

for (String dir : dirs) {
  Path p = Paths.get(dir);
  Files.createDirectories(p);              // fails fast with clear reason
  if (!Files.isWritable(p)) throw new IOException("Not writable: " + p);
}

When it happens

Trigger: Configuring a dfs.datanode.data.dir entry (or hot-swapping a volume in) whose path does not exist and cannot be created: missing parent directory, typo, no mkdir permission, or the underlying mount/device is gone.

Common situations: Typos in data.dir paths; failed or unmounted disk; mount points not yet created when the DN starts; container/k8s volume mounts missing; parent directory read-only for the DN user.

Related errors

• Cannot remove directory {}
• Cannot remove directory {}
• Failed to rename {} to {}
• Invalid path: {location}: directory does not exist
• Storage directory is in use.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/9e153838375ff47e. Report an issue: GitHub.