apache/hadoop · critical · RuntimeException

Cannot start datanode because the configured max locked memo

Error message

Cannot start datanode because the configured max locked memory size (%s) is greater than zero and native code is not available.

What it means

At startup, if dfs.datanode.max.locked.memory is positive (centralized cache management budget), the DN verifies it can lock memory via NativeIO.POSIX.getCacheManipulator().verifyCanMlock(); when native code is unavailable the check fails and the DN aborts with this RuntimeException. Note the message prints the property KEY ('dfs.datanode.max.locked.memory'), not the configured byte count — a long-standing formatting quirk.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java:1898

   * 
   * @param dataDirectories - only for a non-simulated storage data node
   * @throws IOException
   */
  void startDataNode(List<StorageLocation> dataDirectories,
                     SecureResources resources
                     ) throws IOException {

    // settings global for all BPs in the Data Node
    this.secureResources = resources;
    synchronized (this) {
      this.dataDirs = dataDirectories;
    }
    this.dnConf = new DNConf(this);
    checkSecureConfig(dnConf, getConf(), resources);

    if (dnConf.maxLockedMemory > 0) {
      if (!NativeIO.POSIX.getCacheManipulator().verifyCanMlock()) {
        throw new RuntimeException(String.format(
            "Cannot start datanode because the configured max locked memory" +
            " size (%s) is greater than zero and native code is not available.",
            DFS_DATANODE_MAX_LOCKED_MEMORY_KEY));
      }
      if (Path.WINDOWS) {
        NativeIO.Windows.extendWorkingSetSize(dnConf.maxLockedMemory);
      } else {
        long ulimit = NativeIO.POSIX.getCacheManipulator().getMemlockLimit();
        if (dnConf.maxLockedMemory > ulimit) {
          throw new RuntimeException(String.format(
            "Cannot start datanode because the configured max locked memory" +
            " size (%s) of %d bytes is more than the datanode's available" +
            " RLIMIT_MEMLOCK ulimit of %d bytes.",
            DFS_DATANODE_MAX_LOCKED_MEMORY_KEY,
            dnConf.maxLockedMemory,
            ulimit));
        }
      }

View on GitHub (pinned to 2add963021)

Solutions

1. Verify and fix native support: run 'hadoop checknative -a' as the DN user; deploy a matching libhadoop.so and export LD_LIBRARY_PATH/$JAVA_LIBRARY_PATH, then restart
2. If centralized cache management is not needed, set dfs.datanode.max.locked.memory=0 in hdfs-site.xml and restart the DN
3. After native support works, also confirm the memlock ulimit covers the configured bytes before restarting (see the follow-on RLIMIT_MEMLOCK check)

Example fix

<!-- before -->
<property><name>dfs.datanode.max.locked.memory</name><value>4gb</value></property>  <!-- host has no libhadoop.so -->
<!-- after -->
<property><name>dfs.datanode.max.locked.memory</name><value>0</value></property>  <!-- or fix native libs and keep 4gb -->

long maxLocked = conf.getLongBytes(DFS_DATANODE_MAX_LOCKED_MEMORY_KEY, 0);
if (maxLocked > 0 && !NativeIO.POSIX.getCacheManipulator().verifyCanMlock()) {
  // either fix native libs or set max.locked.memory to 0 before starting the DN
  throw new IllegalStateException("max.locked.memory > 0 but mlock unavailable");
}

catch (RuntimeException e) {
  if (e.getMessage().contains("native code is not available")) {
    // run hadoop checknative -a; deploy libhadoop.so; or set dfs.datanode.max.locked.memory=0
  }
}

When it happens

Trigger: dfs.datanode.max.locked.memory > 0 on a JVM whose native Hadoop library did not load: libhadoop.so absent, wrong arch/glibc, LD_LIBRARY_PATH not set, or checknative reports native:false. Only the availability check has run at this point; the ulimit comparison happens after it.

Common situations: Centralized cache management rolled out without shipping native libs; minimal Docker images; JDK or OS upgrades leaving stale native artifacts; deployments that set hadoop.native.lib=false.

Related errors

• Although a UNIX domain socket path is configured as {domainS
• Cluster IDs not matched: dn cid={clusterId} but ns cid={nsCi
• Cannot start datanode because the configured max locked memo
• Invalid value configured for dfs.datanode.failed.volumes.tol
• Security is enabled but block access tokens (via dfs.block.a

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/3d0ff67c2cc80b22. Report an issue: GitHub.