apache/hadoop · error · IOException

Attempt to remove all volumes.

Error message

Attempt to remove all volumes.

What it means

refreshVolumes guards against ending volumeless: it computes numOldDataDirs + getFSDataset().getNumFailedVolumes() + changedVolumes.newLocations.size() - changedVolumes.deactivateLocations.size() and throws IOException("Attempt to remove all volumes.") when the total is <= 0. Because the submitted dfs.datanode.data.dir value is the complete desired set, this fires when every currently active location is dropped and nothing usable is added. Prior parse errors (empty value) are caught earlier by parseChangedVolumes.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java:1306

      nsInfos.add(bpos.getNamespaceInfo());
    }
    synchronized(this) {
      Configuration conf = getConf();
      conf.set(DFS_DATANODE_DATA_DIR_KEY, newVolumes);
      ExecutorService service = null;
      int numOldDataDirs = dataDirs.size();
      ChangedVolumes changedVolumes = parseChangedVolumes(newVolumes);
      StringBuilder errorMessageBuilder = new StringBuilder();
      List<String> effectiveVolumes = Lists.newArrayList();
      for (StorageLocation sl : changedVolumes.unchangedLocations) {
        effectiveVolumes.add(sl.toString());
      }

      try {
        if (numOldDataDirs + getFSDataset().getNumFailedVolumes()
            + changedVolumes.newLocations.size()
            - changedVolumes.deactivateLocations.size() <= 0) {
          throw new IOException("Attempt to remove all volumes.");
        }
        if (!changedVolumes.newLocations.isEmpty()) {
          LOG.info("Adding new volumes: {}",
              Joiner.on(",").join(changedVolumes.newLocations));

          service = Executors
              .newFixedThreadPool(changedVolumes.newLocations.size());
          List<Future<IOException>> exceptions = Lists.newArrayList();

          checkStorageState("refreshVolumes");
          for (final StorageLocation location : changedVolumes.newLocations) {
            exceptions.add(service.submit(new Callable<IOException>() {
              @Override
              public IOException call() {
                try {
                  data.addVolume(location, nsInfos);
                } catch (IOException e) {
                  return e;

View on GitHub (pinned to 2add963021)

Solutions

1. Keep at least one healthy volume in the submitted value — removal means omitting dirs from the list, so list the surviving dirs
2. To retire the last volume, decommission the DataNode instead of live-removing it
3. Before shrinking, check hdfs dfsadmin -reconfig datanode <dn:ipc> status and the DN's NumFailedVolumes JMX

Example fix

# before
dfs.datanode.data.dir=          # or only paths that match nothing
# after
dfs.datanode.data.dir=[DISK]file:///data/dn1,[DISK]file:///data/dn2

int survivors = currentDirs.size() + fsDataset.getNumFailedVolumes()
    + changedVolumes.newLocations.size() - changedVolumes.deactivateLocations.size();
if (survivors <= 0) {
  throw new IllegalArgumentException("refresh would leave zero active volumes; keep at least one");
}

catch (ReconfigurationException e) {
  if (String.valueOf(e.getCause()).contains("Attempt to remove all volumes")) {
    // resubmit with at least one healthy dir in dfs.datanode.data.dir
  }
}

When it happens

Trigger: Live-reconfiguring dfs.datanode.data.dir to a value that omits all currently configured dirs while adding none that stick (e.g. listing only paths that match nothing, or only already-failed volumes); combined with a high failed-volume count on a nearly dead DN.

Common situations: 'Cleanup' scripts that blank or minimize the dir list; attempts to live-decommission a DataNode by removing its last volumes; test environments shrinking storage to zero.

Related errors

• No directory is specified.
• Changing storage type is not allowed.
• Not a valid Boolean value for {property} in reconfSlowPeerPa
• FsDatasetSpi has not been initialized
• Not a valid Boolean value for {property}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/328ffcea7b4a917b. Report an issue: GitHub.