apache/hadoop · error · IOException

StorageLocation specified for storage directory {sd} is null

Error message

StorageLocation specified for storage directory {sd} is null

What it means

FsVolumeImpl's constructor refuses a StorageDirectory whose getStorageLocation() is null: every volume must know the URI it was configured from (parsed from dfs.datanode.data.dir). A null location means storage parsing produced a directory object without a usable location - malformed configuration, cloned/legacy layout, or direct StorageDirectory construction in tests.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java:162

   * Per-volume worker pool that processes new blocks to cache.
   * The maximum number of workers per volume is bounded (configurable via
   * dfs.datanode.fsdatasetcache.max.threads.per.volume) to limit resource
   * contention.
   */
  protected ThreadPoolExecutor cacheExecutor;

  FsVolumeImpl(FsDatasetImpl dataset, String storageID, StorageDirectory sd,
      FileIoProvider fileIoProvider, Configuration conf) throws IOException {
    // outside tests, usage created in ReservedSpaceCalculator.Builder
    this(dataset, storageID, sd, fileIoProvider, conf, null);
  }

  FsVolumeImpl(FsDatasetImpl dataset, String storageID, StorageDirectory sd,
      FileIoProvider fileIoProvider, Configuration conf, DF usage)
      throws IOException {

    if (sd.getStorageLocation() == null) {
      throw new IOException("StorageLocation specified for storage directory " +
          sd + " is null");
    }
    this.dataset = dataset;
    this.storageID = storageID;
    this.reservedForReplicas = new AtomicLong(0L);
    this.storageLocation = sd.getStorageLocation();
    this.currentDir = sd.getCurrentDir();
    this.storageType = storageLocation.getStorageType();
    this.configuredCapacity = -1;
    this.usage = usage;
    if (this.usage != null) {
      reserved = new ReservedSpaceCalculator.Builder(conf)
          .setUsage(this.usage).setStorageType(storageType)
          .setDir(currentDir != null ? currentDir.getParent() : "NULL").build();
      boolean fixedSizeVolume = conf.getBoolean(
          DFSConfigKeys.DFS_DATANODE_FIXED_VOLUME_SIZE_KEY,
          DFSConfigKeys.DFS_DATANODE_FIXED_VOLUME_SIZE_DEFAULT);
      if (fixedSizeVolume) {

View on GitHub (pinned to 2add963021)

Solutions

Check dfs.datanode.data.dir: every entry must be a valid URI such as [disk]file:///dfs/dn or [SSD]file:///... - remove empty or malformed entries.
Verify parsed storage locations in the DN startup log (or hdfs datanode startup text).
If tests build StorageDirectory directly, supply a StorageLocation wrapping a URI when constructing.
Format or roll back cleanly rather than hand-editing in_use.lock/VERSION to point at new dirs.

Defensive patterns

Strategy: validation

Validate before calling

if (sd.getStorageLocation() == null) {
  // reject this storage directory before building a volume
  throw new IOException("StorageDirectory without location: " + sd);
}

Try / catch

try {
  volume = new FsVolumeImpl(dataset, storageID, sd, fileIoProvider, conf);
} catch (IOException e) {
  // skip this directory; DN continues with remaining volumes if tolerated
}

Prevention

Validate every dfs.datanode.data.dir entry as a URI with a storage-type tag before startup.
Never hand-edit VERSION or in_use.lock to redirect storage directories.
In tests, always construct StorageDirectory with an explicit StorageLocation.

When it happens

Trigger: dfs.datanode.data.dir containing an entry that failed URI parsing but still yielded a StorageDirectory; StorageDirectory objects built from a VERSION file without a location (test/utility paths); storage dirs transplanted from a downgrade.

Common situations: Typos or empty entries in dfs.datanode.data.dir; incorrect storage-type tags like [SSD]/[RAM_DISK]; custom tools creating StorageDirectory instances; edge cases after format or rollback.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/9db1a1865971bd2f. Report an issue: GitHub.