apache/hadoop · critical · RuntimeException

Cannot start datanode because the configured max locked memo

Error message

Cannot start datanode because the configured max locked memory size (%s) of %d bytes is more than the datanode's available RLIMIT_MEMLOCK ulimit of %d bytes.

What it means

After the mlock capability check, startDataNode compares dfs.datanode.max.locked.memory with the process RLIMIT_MEMLOCK from NativeIO.POSIX.getCacheManipulator().getMemlockLimit(); if the configured budget exceeds the OS limit the DN aborts at startup. The message interpolates the property key, the configured bytes, and the ulimit bytes, so the shortfall is directly visible.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java:1908

    synchronized (this) {
      this.dataDirs = dataDirectories;
    }
    this.dnConf = new DNConf(this);
    checkSecureConfig(dnConf, getConf(), resources);

    if (dnConf.maxLockedMemory > 0) {
      if (!NativeIO.POSIX.getCacheManipulator().verifyCanMlock()) {
        throw new RuntimeException(String.format(
            "Cannot start datanode because the configured max locked memory" +
            " size (%s) is greater than zero and native code is not available.",
            DFS_DATANODE_MAX_LOCKED_MEMORY_KEY));
      }
      if (Path.WINDOWS) {
        NativeIO.Windows.extendWorkingSetSize(dnConf.maxLockedMemory);
      } else {
        long ulimit = NativeIO.POSIX.getCacheManipulator().getMemlockLimit();
        if (dnConf.maxLockedMemory > ulimit) {
          throw new RuntimeException(String.format(
            "Cannot start datanode because the configured max locked memory" +
            " size (%s) of %d bytes is more than the datanode's available" +
            " RLIMIT_MEMLOCK ulimit of %d bytes.",
            DFS_DATANODE_MAX_LOCKED_MEMORY_KEY,
            dnConf.maxLockedMemory,
            ulimit));
        }
      }
    }
    LOG.info("Starting DataNode with maxLockedMemory = {}",
        dnConf.maxLockedMemory);

    int volFailuresTolerated = dnConf.getVolFailuresTolerated();
    int volsConfigured = dnConf.getVolsConfigured();
    if (volFailuresTolerated < MAX_VOLUME_FAILURE_TOLERATED_LIMIT
        || volFailuresTolerated >= volsConfigured) {
      throw new HadoopIllegalArgumentException("Invalid value configured for "
          + "dfs.datanode.failed.volumes.tolerated - " + volFailuresTolerated

View on GitHub (pinned to 2add963021)

Solutions

1. Raise the memlock limit for the DN process: /etc/security/limits.conf 'hdfs soft/hard memlock unlimited', or LimitMEMLOCK=infinity in the systemd unit, then fully restart the service
2. Or shrink dfs.datanode.max.locked.memory to fit under the current ulimit (the value is bytes; size suffixes like 4gb are accepted)
3. Verify at runtime: cat /proc/$(pidof DataNode)/limits | grep -i memlock shows the effective ceiling

Example fix

# before
# /etc/security/limits.conf: (nothing) -> ulimit -l 65536 bytes
dfs.datanode.max.locked.memory=4gb
# after
# /etc/security/limits.conf
hdfs soft memlock unlimited
hdfs hard memlock unlimited
# systemd unit: LimitMEMLOCK=infinity, then restart the DataNode

long maxLocked = conf.getLongBytes(DFS_DATANODE_MAX_LOCKED_MEMORY_KEY, 0);
long memlock = NativeIO.POSIX.getCacheManipulator().getMemlockLimit();
if (maxLocked > memlock) {
  throw new IllegalStateException("maxLockedMemory " + maxLocked + " > RLIMIT_MEMLOCK " + memlock
      + " — raise ulimit -l / LimitMEMLOCK or lower the setting");
}

catch (RuntimeException e) {
  if (e.getMessage().contains("RLIMIT_MEMLOCK")) {
    // raise memlock limit for the hdfs service user and restart, or shrink the configured budget
  }
}

When it happens

Trigger: dfs.datanode.max.locked.memory greater than 'ulimit -l' for the hdfs user: common defaults of 64 KB versus multi-GB cache budgets; systemd-managed services without LimitMEMLOCK raised; /etc/security/limits.conf applied to login shells but not to the daemon's launch context.

Common situations: Centralized cache rollouts on default-locked-down hosts; containers/cgroups with hard memlock caps; PAM limits not applied because the daemon is started by systemd rather than a login session.

Related errors

• Cannot start datanode because the configured max locked memo
• Not a valid Boolean value for {property} in reconfSlowPeerPa
• FsDatasetSpi has not been initialized
• Not a valid Boolean value for {property}
• No directory is specified.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/9cbefc811f925b69. Report an issue: GitHub.