{"record":{"id":"3d0ff67c2cc80b22","repo":"apache/hadoop","slug":"cannot-start-datanode-because-the-configured-max-l","errorCode":null,"errorMessage":"Cannot start datanode because the configured max locked memory size (%s) is greater than zero and native code is not available.","messagePattern":"Cannot start datanode because the configured max locked memory size \\((.+?)\\) is greater than zero and native code is not available\\.","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"critical","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java","lineNumber":1898,"sourceCode":"   * \n   * @param dataDirectories - only for a non-simulated storage data node\n   * @throws IOException\n   */\n  void startDataNode(List<StorageLocation> dataDirectories,\n                     SecureResources resources\n                     ) throws IOException {\n\n    // settings global for all BPs in the Data Node\n    this.secureResources = resources;\n    synchronized (this) {\n      this.dataDirs = dataDirectories;\n    }\n    this.dnConf = new DNConf(this);\n    checkSecureConfig(dnConf, getConf(), resources);\n\n    if (dnConf.maxLockedMemory > 0) {\n      if (!NativeIO.POSIX.getCacheManipulator().verifyCanMlock()) {\n        throw new RuntimeException(String.format(\n            \"Cannot start datanode because the configured max locked memory\" +\n            \" size (%s) is greater than zero and native code is not available.\",\n            DFS_DATANODE_MAX_LOCKED_MEMORY_KEY));\n      }\n      if (Path.WINDOWS) {\n        NativeIO.Windows.extendWorkingSetSize(dnConf.maxLockedMemory);\n      } else {\n        long ulimit = NativeIO.POSIX.getCacheManipulator().getMemlockLimit();\n        if (dnConf.maxLockedMemory > ulimit) {\n          throw new RuntimeException(String.format(\n            \"Cannot start datanode because the configured max locked memory\" +\n            \" size (%s) of %d bytes is more than the datanode's available\" +\n            \" RLIMIT_MEMLOCK ulimit of %d bytes.\",\n            DFS_DATANODE_MAX_LOCKED_MEMORY_KEY,\n            dnConf.maxLockedMemory,\n            ulimit));\n        }\n      }","sourceCodeStart":1880,"sourceCodeEnd":1916,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L1880-L1916","documentation":"At startup, if dfs.datanode.max.locked.memory is positive (centralized cache management budget), the DN verifies it can lock memory via NativeIO.POSIX.getCacheManipulator().verifyCanMlock(); when native code is unavailable the check fails and the DN aborts with this RuntimeException. Note the message prints the property KEY ('dfs.datanode.max.locked.memory'), not the configured byte count — a long-standing formatting quirk.","triggerScenarios":"dfs.datanode.max.locked.memory > 0 on a JVM whose native Hadoop library did not load: libhadoop.so absent, wrong arch/glibc, LD_LIBRARY_PATH not set, or checknative reports native:false. Only the availability check has run at this point; the ulimit comparison happens after it.","commonSituations":"Centralized cache management rolled out without shipping native libs; minimal Docker images; JDK or OS upgrades leaving stale native artifacts; deployments that set hadoop.native.lib=false.","solutions":["Verify and fix native support: run 'hadoop checknative -a' as the DN user; deploy a matching libhadoop.so and export LD_LIBRARY_PATH/$JAVA_LIBRARY_PATH, then restart","If centralized cache management is not needed, set dfs.datanode.max.locked.memory=0 in hdfs-site.xml and restart the DN","After native support works, also confirm the memlock ulimit covers the configured bytes before restarting (see the follow-on RLIMIT_MEMLOCK check)"],"exampleFix":"<!-- before -->\n<property><name>dfs.datanode.max.locked.memory</name><value>4gb</value></property>  <!-- host has no libhadoop.so -->\n<!-- after -->\n<property><name>dfs.datanode.max.locked.memory</name><value>0</value></property>  <!-- or fix native libs and keep 4gb -->","handlingStrategy":"validation","validationCode":"long maxLocked = conf.getLongBytes(DFS_DATANODE_MAX_LOCKED_MEMORY_KEY, 0);\nif (maxLocked > 0 && !NativeIO.POSIX.getCacheManipulator().verifyCanMlock()) {\n  // either fix native libs or set max.locked.memory to 0 before starting the DN\n  throw new IllegalStateException(\"max.locked.memory > 0 but mlock unavailable\");\n}","typeGuard":null,"tryCatchPattern":"catch (RuntimeException e) {\n  if (e.getMessage().contains(\"native code is not available\")) {\n    // run hadoop checknative -a; deploy libhadoop.so; or set dfs.datanode.max.locked.memory=0\n  }\n}","preventionTips":["Gate centralized-cache rollouts on a passed 'hadoop checknative -a' per host","Keep native libs in deployment images; re-verify after glibc/JDK/arch upgrades","If a host cannot support mlock, leave dfs.datanode.max.locked.memory unset (0) there"],"tags":["hadoop","hdfs","datanode","native-library","centralized-cache","startup"],"backgroundTag":"native-library-load-failed","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}