apache/hadoop · error · IOException

Storage not yet initialized for {}

Error message

Storage not yet initialized for {}

What it means

DataNode.checkStorageState guards every volume/block-pool RPC: if the FsDatasetSpi field 'data' is still null, the DataNode has not finished (or has failed) storage initialization, so the call is rejected with an IOException naming the caller method. It means the datanode process is alive but not yet able to serve storage operations.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java:3871

          blockPoolId);
      throw new IOException(
          "The block pool is still running. First do a refreshNamenodes to " +
          "shutdown the block pool service");
    }
    checkStorageState("deleteBlockPool");
    data.deleteBlockPool(blockPoolId, force);
  }

  /**
   * Check if storage has been initialized.
   * @param methodName caller name
   * @throws IOException throw IOException if not yet initialized.
   */
  private void checkStorageState(String methodName) throws IOException {
    if (data == null) {
      String message = "Storage not yet initialized for " + methodName;
      LOG.debug(message);
      throw new IOException(message);
    }
  }

  @Override // ClientDatanodeProtocol
  public synchronized void shutdownDatanode(boolean forUpgrade) throws IOException {
    checkSuperuserPrivilege();
    LOG.info("shutdownDatanode command received (upgrade={}). " +
        "Shutting down Datanode...", forUpgrade);

    // Shutdown can be called only once.
    if (shutdownInProgress) {
      throw new IOException("Shutdown already in progress.");
    }
    shutdownInProgress = true;
    shutdownForUpgrade = forUpgrade;

    // Asynchronously start the shutdown process so that the rpc response can be
    // sent back.

View on GitHub (pinned to 2add963021)

Solutions

1. Wait for full DN startup (log line 'DataNode is completely started' or JMX DatanodeInfo showing registration) before issuing volume RPCs
2. Retry the RPC with backoff until storage initialization completes
3. Check the DN log for the earlier real failure (e.g. 'All specified directories have failed to load.') and fix that first
4. Verify at least one valid, writable dfs.datanode.data.dir is configured

Example fix

// before
clientDatanodeProtocol.refreshVolumes(newLocations); // fires during DN startup -> IOException

// after
for (int i = 0; i < 30; i++) {
  try {
    clientDatanodeProtocol.refreshVolumes(newLocations);
    break;
  } catch (IOException e) {
    if (e.getMessage().contains("Storage not yet initialized")) {
      Thread.sleep(1000); // DN still starting; back off and retry
      continue;
    }
    throw e;
  }
}

// Before issuing volume RPCs, confirm the DN is fully up (JMX bean registered)
// e.g. query http://<dn>:<jmxport>/jmx for Hadoop:service=DataNode,name=DataNodeInfo
// and check it reports a DatanodeRegistration / started state.

catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().contains("Storage not yet initialized")) {
    // DN still starting: back off and retry the RPC
  } else {
    throw e;
  }
}

When it happens

Trigger: Invoking ClientDatanodeProtocol operations that route through checkStorageState (e.g. refreshVolumes, getVolumeReport, deleteBlockPool) against a datanode whose startDataNode/instantiateDataNode has not completed, or whose storage layer failed to initialize (data == null).

Common situations: Automation or dfsadmin scripts firing RPCs immediately after launching the DataNode process; retry loops racing DN startup; a DN whose disk setup failed earlier so 'data' was never assigned.

Related errors

• FsDatasetSpi has not been initialized
• Cluster IDs not matched: dn cid={clusterId} but ns cid={nsCi
• Although a UNIX domain socket path is configured as {domainS
• Cannot start datanode because the configured max locked memo
• Invalid value configured for dfs.datanode.failed.volumes.tol

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/fac45ad8b4ed4046. Report an issue: GitHub.