{"record":{"id":"328ffcea7b4a917b","repo":"apache/hadoop","slug":"attempt-to-remove-all-volumes","errorCode":null,"errorMessage":"Attempt to remove all volumes.","messagePattern":"Attempt to remove all volumes\\.","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java","lineNumber":1306,"sourceCode":"      nsInfos.add(bpos.getNamespaceInfo());\n    }\n    synchronized(this) {\n      Configuration conf = getConf();\n      conf.set(DFS_DATANODE_DATA_DIR_KEY, newVolumes);\n      ExecutorService service = null;\n      int numOldDataDirs = dataDirs.size();\n      ChangedVolumes changedVolumes = parseChangedVolumes(newVolumes);\n      StringBuilder errorMessageBuilder = new StringBuilder();\n      List<String> effectiveVolumes = Lists.newArrayList();\n      for (StorageLocation sl : changedVolumes.unchangedLocations) {\n        effectiveVolumes.add(sl.toString());\n      }\n\n      try {\n        if (numOldDataDirs + getFSDataset().getNumFailedVolumes()\n            + changedVolumes.newLocations.size()\n            - changedVolumes.deactivateLocations.size() <= 0) {\n          throw new IOException(\"Attempt to remove all volumes.\");\n        }\n        if (!changedVolumes.newLocations.isEmpty()) {\n          LOG.info(\"Adding new volumes: {}\",\n              Joiner.on(\",\").join(changedVolumes.newLocations));\n\n          service = Executors\n              .newFixedThreadPool(changedVolumes.newLocations.size());\n          List<Future<IOException>> exceptions = Lists.newArrayList();\n\n          checkStorageState(\"refreshVolumes\");\n          for (final StorageLocation location : changedVolumes.newLocations) {\n            exceptions.add(service.submit(new Callable<IOException>() {\n              @Override\n              public IOException call() {\n                try {\n                  data.addVolume(location, nsInfos);\n                } catch (IOException e) {\n                  return e;","sourceCodeStart":1288,"sourceCodeEnd":1324,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L1288-L1324","documentation":"refreshVolumes guards against ending volumeless: it computes numOldDataDirs + getFSDataset().getNumFailedVolumes() + changedVolumes.newLocations.size() - changedVolumes.deactivateLocations.size() and throws IOException(\"Attempt to remove all volumes.\") when the total is <= 0. Because the submitted dfs.datanode.data.dir value is the complete desired set, this fires when every currently active location is dropped and nothing usable is added. Prior parse errors (empty value) are caught earlier by parseChangedVolumes.","triggerScenarios":"Live-reconfiguring dfs.datanode.data.dir to a value that omits all currently configured dirs while adding none that stick (e.g. listing only paths that match nothing, or only already-failed volumes); combined with a high failed-volume count on a nearly dead DN.","commonSituations":"'Cleanup' scripts that blank or minimize the dir list; attempts to live-decommission a DataNode by removing its last volumes; test environments shrinking storage to zero.","solutions":["Keep at least one healthy volume in the submitted value — removal means omitting dirs from the list, so list the surviving dirs","To retire the last volume, decommission the DataNode instead of live-removing it","Before shrinking, check hdfs dfsadmin -reconfig datanode <dn:ipc> status and the DN's NumFailedVolumes JMX"],"exampleFix":"# before\ndfs.datanode.data.dir=          # or only paths that match nothing\n# after\ndfs.datanode.data.dir=[DISK]file:///data/dn1,[DISK]file:///data/dn2","handlingStrategy":"validation","validationCode":"int survivors = currentDirs.size() + fsDataset.getNumFailedVolumes()\n    + changedVolumes.newLocations.size() - changedVolumes.deactivateLocations.size();\nif (survivors <= 0) {\n  throw new IllegalArgumentException(\"refresh would leave zero active volumes; keep at least one\");\n}","typeGuard":null,"tryCatchPattern":"catch (ReconfigurationException e) {\n  if (String.valueOf(e.getCause()).contains(\"Attempt to remove all volumes\")) {\n    // resubmit with at least one healthy dir in dfs.datanode.data.dir\n  }\n}","preventionTips":["Always keep at least one healthy volume in the submitted value","Decommission the DN instead of live-removing its final volume","Include NumFailedVolumes in pre-flight math when volumes are degraded"],"tags":["hadoop","hdfs","datanode","volume-management","reconfiguration","guard-clause"],"backgroundTag":"remove-all-volumes-blocked","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}