apache/hadoop · error · IOException

FsDatasetSpi has not been initialized

Error message

FsDatasetSpi has not been initialized

What it means

reconfDfsUsageParameters live-applies fs.du.interval and fs.getspaceused.classname by pushing new settings into every FsVolumeImpl's BlockPoolSlice; it requires the DataNode's FsDatasetSpi (the 'data' field) to exist. If the reconfig request lands while data == null — DN still starting, dataset failed to build, or already shut down — it throws IOException("FsDatasetSpi has not been initialized"), which the method's own catch then wraps into a ReconfigurationException.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java:976

            DFS_DATANODE_MAX_SLOWDISKS_TO_EXCLUDE_DEFAULT : Integer.parseInt(newVal));
        result = Integer.toString(maxSlowDisksToExclude);
        diskMetrics.setMaxSlowDisksToExclude(maxSlowDisksToExclude);
      }
      LOG.info("RECONFIGURE* changed {} to {}", property, newVal);
      return result;
    } catch (IllegalArgumentException e) {
      throw new ReconfigurationException(property, newVal, getConf().get(property), e);
    }
  }

  private String reconfDfsUsageParameters(String property, String newVal)
      throws ReconfigurationException {
    String result = null;
    try {
      LOG.info("Reconfiguring {} to {}", property, newVal);
      if (data == null) {
        LOG.debug("FsDatasetSpi has not been initialized.");
        throw new IOException("FsDatasetSpi has not been initialized");
      }
      if (property.equals(FS_DU_INTERVAL_KEY)) {
        long interval = (newVal == null ? FS_DU_INTERVAL_DEFAULT :
            Long.parseLong(newVal));
        result = Long.toString(interval);
        List<FsVolumeImpl> volumeList = data.getVolumeList();
        for (FsVolumeImpl fsVolume : volumeList) {
          Map<String, BlockPoolSlice> blockPoolSlices = fsVolume.getBlockPoolSlices();
          for (BlockPoolSlice value : blockPoolSlices.values()) {
            value.updateDfsUsageConfig(interval, null, null);
          }
        }
      } else if (property.equals(FS_GETSPACEUSED_JITTER_KEY)) {
        long jitter = (newVal == null ? FS_GETSPACEUSED_JITTER_DEFAULT :
            Long.parseLong(newVal));
        result = Long.toString(jitter);
        List<FsVolumeImpl> volumeList = data.getVolumeList();
        for (FsVolumeImpl fsVolume : volumeList) {

View on GitHub (pinned to 2add963021)

Solutions

1. Wait until the DN reports to the NameNode (hdfs dfsadmin -report shows it live) and resubmit the reconfiguration
2. Inspect DN logs for the underlying FsDataset creation failure (permissions, incompatible VERSION, failed volumes) and fix that first
3. For fs.du.interval / fs.getspaceused.classname prefer static hdfs-site.xml changes plus a rolling restart over live reconfig

// Only submit usage reconfigs once the DN is fully up:
// 1) 'hdfs dfsadmin -report' lists the DN as live, or
// 2) curl http://dn:9864/jmx shows DataNodeActivity beans with heartbeat counters > 0

try {
  dn.reconfigurePropertyImpl(key, newVal); // fs.du.interval / fs.getspaceused.classname
} catch (ReconfigurationException e) {
  if (e.getCause() instanceof IOException
      && String.valueOf(e.getCause().getMessage()).contains("FsDatasetSpi")) {
    // lifecycle race: DN starting or shutting down — retry after it reports to the NN
  }
}

When it happens

Trigger: Reconfiguring fs.du.interval or fs.getspaceused.classname immediately after process start (racing volume formatting/fencing), on a DN whose FsDataset failed to initialize (bad permissions, failed volumes), or during shutdown while a reconfig driver keeps targeting the node.

Common situations: Automation fires reconfig calls as soon as the DN port opens instead of after NN registration; DN startup aborted half-way leaving data null; reconfig storms during rolling upgrades hitting a node mid-restart.

Related errors

• Not a valid Boolean value for {property} in reconfSlowPeerPa
• Not a valid Boolean value for {property}
• No directory is specified.
• Changing storage type is not allowed.
• Attempt to remove all volumes.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/a3fadaa76cd128dd. Report an issue: GitHub.