apache/hadoop · critical · HadoopIllegalArgumentException

Invalid value configured for dfs.datanode.failed.volumes.tol

Error message

Invalid value configured for dfs.datanode.failed.volumes.tolerated - {volFailuresTolerated}. Value configured is either less than -1 or >= to the number of configured volumes ({volsConfigured}).

What it means

startDataNode validates dfs.datanode.failed.volumes.tolerated against the configured dir count: the value must be >= MAX_VOLUME_FAILURE_TOLERATED_LIMIT (-1) and strictly less than volsConfigured; anything else throws HadoopIllegalArgumentException and the DN exits. The message shows both the tolerated value and the configured volume count so the bad combination is obvious.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java:1925

        if (dnConf.maxLockedMemory > ulimit) {
          throw new RuntimeException(String.format(
            "Cannot start datanode because the configured max locked memory" +
            " size (%s) of %d bytes is more than the datanode's available" +
            " RLIMIT_MEMLOCK ulimit of %d bytes.",
            DFS_DATANODE_MAX_LOCKED_MEMORY_KEY,
            dnConf.maxLockedMemory,
            ulimit));
        }
      }
    }
    LOG.info("Starting DataNode with maxLockedMemory = {}",
        dnConf.maxLockedMemory);

    int volFailuresTolerated = dnConf.getVolFailuresTolerated();
    int volsConfigured = dnConf.getVolsConfigured();
    if (volFailuresTolerated < MAX_VOLUME_FAILURE_TOLERATED_LIMIT
        || volFailuresTolerated >= volsConfigured) {
      throw new HadoopIllegalArgumentException("Invalid value configured for "
          + "dfs.datanode.failed.volumes.tolerated - " + volFailuresTolerated
          + ". Value configured is either less than -1 or >= "
          + "to the number of configured volumes (" + volsConfigured + ").");
    }

    storage = new DataStorage();
    
    // global DN settings
    registerMXBean();
    initDataXceiver();
    startInfoServer();
    pauseMonitor = new JvmPauseMonitor();
    pauseMonitor.init(getConf());
    pauseMonitor.start();
  
    // BlockPoolTokenSecretManager is required to create ipc server.
    this.blockPoolTokenSecretManager = new BlockPoolTokenSecretManager();

View on GitHub (pinned to 2add963021)

Solutions

1. Set dfs.datanode.failed.volumes.tolerated within [-1, #dirs - 1]; on a 1-dir node use 0
2. Express the setting in templates as a function of the dir list length instead of a constant
3. Correct hdfs-site.xml and restart the DataNode

Example fix

<!-- before: single volume, tolerated=2 copied from a 6-disk node -->
<property><name>dfs.datanode.failed.volumes.tolerated</name><value>2</value></property>
<!-- after -->
<property><name>dfs.datanode.failed.volumes.tolerated</name><value>0</value></property>

int tolerated = conf.getInt(DFS_DATANODE_FAILED_VOLUMES_KEY, 0);
int dirs = DataNode.getStorageLocations(conf).size();
if (tolerated < -1 || tolerated >= dirs) {
  throw new IllegalArgumentException("failed.volumes.tolerated must be in [-1, " + (dirs - 1)
      + "] for " + dirs + " configured dirs");
}

catch (HadoopIllegalArgumentException e) {
  // message shows tolerated vs configured counts; fix hdfs-site.xml and restart the DN
}

When it happens

Trigger: A single-volume DataNode with tolerated >= 1 (template defaults like 2 fail: 2 >= 1 dir); values < -1 such as -2; shrinking dfs.datanode.data.dir from 6 dirs to 1 without lowering the tolerated count; tolerated exactly equal to the number of dirs.

Common situations: Test or lab nodes with one disk inheriting production defaults; automation that scales down dir counts but not the tolerated count; fresh installs copying a big-cluster hdfs-site.xml.

Related errors

• No directory is specified.
• Security is enabled but block access tokens (via dfs.block.a
• Incompatible node types: storageType={storageType} but Stora
• Not a valid Boolean value for {property} in reconfSlowPeerPa
• Not a valid Boolean value for {property}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/ca75a1b92ca674a5. Report an issue: GitHub.