{"record":{"id":"7418b5ff924f1d60","repo":"apache/hadoop","slug":"too-many-failed-volumes-current-valid-volumes","errorCode":null,"errorMessage":"Too many failed volumes - current valid volumes: {}, volumes configured: {}, volumes failed: {}, volume failures tolerated: {}","messagePattern":"Too many failed volumes - current valid volumes: (.+?), volumes configured: (.+?), volumes failed: (.+?), volume failures tolerated: (.+?)","errorType":"exception","errorClass":"DiskErrorException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/checker/StorageLocationChecker.java","lineNumber":222,"sourceCode":"          LOG.warn(\"StorageLocation {} detected as failed.\", location);\n          failedLocations.add(location);\n          goodLocations.remove(location);\n          break;\n        default:\n          LOG.error(\"Unexpected health check result {} for StorageLocation {}\",\n              result, location);\n        }\n      } catch (ExecutionException|TimeoutException e) {\n        LOG.warn(\"Exception checking StorageLocation \" + location,\n            e.getCause());\n        failedLocations.add(location);\n        goodLocations.remove(location);\n      }\n    }\n\n    if (maxVolumeFailuresTolerated == DataNode.MAX_VOLUME_FAILURE_TOLERATED_LIMIT) {\n      if (dataDirs.size() == failedLocations.size()) {\n        throw new DiskErrorException(\"Too many failed volumes - \"\n            + \"current valid volumes: \" + goodLocations.size()\n            + \", volumes configured: \" + dataDirs.size()\n            + \", volumes failed: \" + failedLocations.size()\n            + \", volume failures tolerated: \" + maxVolumeFailuresTolerated);\n      }\n    } else {\n      if (failedLocations.size() > maxVolumeFailuresTolerated) {\n        throw new DiskErrorException(\"Too many failed volumes - \"\n            + \"current valid volumes: \" + goodLocations.size()\n            + \", volumes configured: \" + dataDirs.size()\n            + \", volumes failed: \" + failedLocations.size()\n            + \", volume failures tolerated: \" + maxVolumeFailuresTolerated);\n      }\n    }\n\n    if (goodLocations.size() == 0) {\n      throw new DiskErrorException(\"All directories in \"\n          + DFS_DATANODE_DATA_DIR_KEY + \" are invalid: \"","sourceCodeStart":204,"sourceCodeEnd":240,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/checker/StorageLocationChecker.java#L204-L240","documentation":"Error \"Too many failed volumes - current valid volumes: {}, volumes configured: {}, volumes failed: {}, volume failures tolerated: {}\" thrown in apache/hadoop.","triggerScenarios":"DataNode startup or storage check when the number of failed data volumes exceeds dfs.datanode.failed.volumes.tolerated. Avoid by replacing failed disks or raising the tolerated-failure count.","commonSituations":"See trigger scenarios.","solutions":["Check the health of the configured data volumes: run 'df -h' and 'dmesg' on the DataNode host to find failed or full disks listed in dfs.datanode.data.dir.","Replace or remount the failed disks, then restart the DataNode.","If transient failures are acceptable, raise dfs.datanode.failed.volumes.tolerated so the DataNode stays up with fewer valid volumes.","Fix filesystem permissions on the configured volume directories so the DataNode user can read and write them."],"exampleFix":null,"handlingStrategy":null,"validationCode":null,"typeGuard":null,"tryCatchPattern":null,"preventionTips":[],"tags":[],"backgroundTag":null,"analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}