apache/hadoop · critical · HadoopIllegalArgumentException

Invalid value configured for…

Error message

Invalid value configured for dfs.datanode.failed.volumes.tolerated - {volFailuresTolerated}. Value configured is either less than -1 or >= to the number of configured volumes ({volsConfigured}).

What it means

startDataNode validates dfs.datanode.failed.volumes.tolerated against the configured dir count: the value must be >= MAX_VOLUME_FAILURE_TOLERATED_LIMIT (-1) and strictly less than volsConfigured; anything else throws HadoopIllegalArgumentException and the DN exits. The message shows both the tolerated value and the configured volume count so the bad combination is obvious.

Solutions

1. Set dfs.datanode.failed.volumes.tolerated within [-1, #dirs - 1]; on a 1-dir node use 0
2. Express the setting in templates as a function of the dir list length instead of a constant
3. Correct hdfs-site.xml and restart the DataNode

Example fix

<!-- before: single volume, tolerated=2 copied from a 6-disk node -->
<property><name>dfs.datanode.failed.volumes.tolerated</name><value>2</value></property>
<!-- after -->
<property><name>dfs.datanode.failed.volumes.tolerated</name><value>0</value></property>

int tolerated = conf.getInt(DFS_DATANODE_FAILED_VOLUMES_KEY, 0);
int dirs = DataNode.getStorageLocations(conf).size();
if (tolerated < -1 || tolerated >= dirs) {
  throw new IllegalArgumentException("failed.volumes.tolerated must be in [-1, " + (dirs - 1)
      + "] for " + dirs + " configured dirs");
}

catch (HadoopIllegalArgumentException e) {
  // message shows tolerated vs configured counts; fix hdfs-site.xml and restart the DN
}

When it happens

Trigger: A single-volume DataNode with tolerated >= 1 (template defaults like 2 fail: 2 >= 1 dir); values < -1 such as -2; shrinking dfs.datanode.data.dir from 6 dirs to 1 without lowering the tolerated count; tolerated exactly equal to the number of dirs.

Common situations: Test or lab nodes with one disk inheriting production defaults; automation that scales down dir counts but not the tolerated count; fresh installs copying a big-cluster hdfs-site.xml.

Understand the failure class

Background: "Invalid value" and "allowed values are" config errors: what your library rejected and how to fix it — this error's family across 41 libraries.

Related errors

• No directory is specified.
• Security is enabled but block access tokens (via…
• All specified directories have failed to load.
• Although a UNIX domain socket path is configured as
• Attempt to remove all volumes.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/ca75a1b92ca674a5. Report an issue: GitHub.

Appendix: source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java:1925

        if (dnConf.maxLockedMemory > ulimit) {
          throw new RuntimeException(String.format(
            "Cannot start datanode because the configured max locked memory" +
            " size (%s) of %d bytes is more than the datanode's available" +
            " RLIMIT_MEMLOCK ulimit of %d bytes.",
            DFS_DATANODE_MAX_LOCKED_MEMORY_KEY,
            dnConf.maxLockedMemory,
            ulimit));
        }
      }
    }
    LOG.info("Starting DataNode with maxLockedMemory = {}",
        dnConf.maxLockedMemory);

    int volFailuresTolerated = dnConf.getVolFailuresTolerated();
    int volsConfigured = dnConf.getVolsConfigured();
    if (volFailuresTolerated < MAX_VOLUME_FAILURE_TOLERATED_LIMIT
        || volFailuresTolerated >= volsConfigured) {
      throw new HadoopIllegalArgumentException("Invalid value configured for "
          + "dfs.datanode.failed.volumes.tolerated - " + volFailuresTolerated
          + ". Value configured is either less than -1 or >= "
          + "to the number of configured volumes (" + volsConfigured + ").");
    }

    storage = new DataStorage();
    
    // global DN settings
    registerMXBean();
    initDataXceiver();
    startInfoServer();
    pauseMonitor = new JvmPauseMonitor();
    pauseMonitor.init(getConf());
    pauseMonitor.start();
  
    // BlockPoolTokenSecretManager is required to create ipc server.
    this.blockPoolTokenSecretManager = new BlockPoolTokenSecretManager();

View on GitHub (pinned to 2add963021)