apache/hadoop · error · DiskErrorException

Too many failed volumes - current valid volumes: {}, volumes

Error message

Too many failed volumes - current valid volumes: {}, volumes configured: {}, volumes failed: {}, volume failures tolerated: {}

What it means

Error "Too many failed volumes - current valid volumes: {}, volumes configured: {}, volumes failed: {}, volume failures tolerated: {}" thrown in apache/hadoop.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/checker/StorageLocationChecker.java:222

          LOG.warn("StorageLocation {} detected as failed.", location);
          failedLocations.add(location);
          goodLocations.remove(location);
          break;
        default:
          LOG.error("Unexpected health check result {} for StorageLocation {}",
              result, location);
        }
      } catch (ExecutionException|TimeoutException e) {
        LOG.warn("Exception checking StorageLocation " + location,
            e.getCause());
        failedLocations.add(location);
        goodLocations.remove(location);
      }
    }

    if (maxVolumeFailuresTolerated == DataNode.MAX_VOLUME_FAILURE_TOLERATED_LIMIT) {
      if (dataDirs.size() == failedLocations.size()) {
        throw new DiskErrorException("Too many failed volumes - "
            + "current valid volumes: " + goodLocations.size()
            + ", volumes configured: " + dataDirs.size()
            + ", volumes failed: " + failedLocations.size()
            + ", volume failures tolerated: " + maxVolumeFailuresTolerated);
      }
    } else {
      if (failedLocations.size() > maxVolumeFailuresTolerated) {
        throw new DiskErrorException("Too many failed volumes - "
            + "current valid volumes: " + goodLocations.size()
            + ", volumes configured: " + dataDirs.size()
            + ", volumes failed: " + failedLocations.size()
            + ", volume failures tolerated: " + maxVolumeFailuresTolerated);
      }
    }

    if (goodLocations.size() == 0) {
      throw new DiskErrorException("All directories in "
          + DFS_DATANODE_DATA_DIR_KEY + " are invalid: "

View on GitHub (pinned to 2add963021)

Solutions

1. Check the health of the configured data volumes: run 'df -h' and 'dmesg' on the DataNode host to find failed or full disks listed in dfs.datanode.data.dir.
2. Replace or remount the failed disks, then restart the DataNode.
3. If transient failures are acceptable, raise dfs.datanode.failed.volumes.tolerated so the DataNode stays up with fewer valid volumes.
4. Fix filesystem permissions on the configured volume directories so the DataNode user can read and write them.

When it happens

Trigger: DataNode startup or storage check when the number of failed data volumes exceeds dfs.datanode.failed.volumes.tolerated. Avoid by replacing failed disks or raising the tolerated-failure count.

Common situations: See trigger scenarios.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/7418b5ff924f1d60. Report an issue: GitHub.