{"record":{"id":"9cbefc811f925b69","repo":"apache/hadoop","slug":"cannot-start-datanode-because-the-configured-max-l-9cbefc","errorCode":null,"errorMessage":"Cannot start datanode because the configured max locked memory size (%s) of %d bytes is more than the datanode's available RLIMIT_MEMLOCK ulimit of %d bytes.","messagePattern":"Cannot start datanode because the configured max locked memory size \\((.+?)\\) of (.+?) bytes is more than the datanode's available RLIMIT_MEMLOCK ulimit of (.+?) bytes\\.","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"critical","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java","lineNumber":1908,"sourceCode":"    synchronized (this) {\n      this.dataDirs = dataDirectories;\n    }\n    this.dnConf = new DNConf(this);\n    checkSecureConfig(dnConf, getConf(), resources);\n\n    if (dnConf.maxLockedMemory > 0) {\n      if (!NativeIO.POSIX.getCacheManipulator().verifyCanMlock()) {\n        throw new RuntimeException(String.format(\n            \"Cannot start datanode because the configured max locked memory\" +\n            \" size (%s) is greater than zero and native code is not available.\",\n            DFS_DATANODE_MAX_LOCKED_MEMORY_KEY));\n      }\n      if (Path.WINDOWS) {\n        NativeIO.Windows.extendWorkingSetSize(dnConf.maxLockedMemory);\n      } else {\n        long ulimit = NativeIO.POSIX.getCacheManipulator().getMemlockLimit();\n        if (dnConf.maxLockedMemory > ulimit) {\n          throw new RuntimeException(String.format(\n            \"Cannot start datanode because the configured max locked memory\" +\n            \" size (%s) of %d bytes is more than the datanode's available\" +\n            \" RLIMIT_MEMLOCK ulimit of %d bytes.\",\n            DFS_DATANODE_MAX_LOCKED_MEMORY_KEY,\n            dnConf.maxLockedMemory,\n            ulimit));\n        }\n      }\n    }\n    LOG.info(\"Starting DataNode with maxLockedMemory = {}\",\n        dnConf.maxLockedMemory);\n\n    int volFailuresTolerated = dnConf.getVolFailuresTolerated();\n    int volsConfigured = dnConf.getVolsConfigured();\n    if (volFailuresTolerated < MAX_VOLUME_FAILURE_TOLERATED_LIMIT\n        || volFailuresTolerated >= volsConfigured) {\n      throw new HadoopIllegalArgumentException(\"Invalid value configured for \"\n          + \"dfs.datanode.failed.volumes.tolerated - \" + volFailuresTolerated","sourceCodeStart":1890,"sourceCodeEnd":1926,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L1890-L1926","documentation":"After the mlock capability check, startDataNode compares dfs.datanode.max.locked.memory with the process RLIMIT_MEMLOCK from NativeIO.POSIX.getCacheManipulator().getMemlockLimit(); if the configured budget exceeds the OS limit the DN aborts at startup. The message interpolates the property key, the configured bytes, and the ulimit bytes, so the shortfall is directly visible.","triggerScenarios":"dfs.datanode.max.locked.memory greater than 'ulimit -l' for the hdfs user: common defaults of 64 KB versus multi-GB cache budgets; systemd-managed services without LimitMEMLOCK raised; /etc/security/limits.conf applied to login shells but not to the daemon's launch context.","commonSituations":"Centralized cache rollouts on default-locked-down hosts; containers/cgroups with hard memlock caps; PAM limits not applied because the daemon is started by systemd rather than a login session.","solutions":["Raise the memlock limit for the DN process: /etc/security/limits.conf 'hdfs soft/hard memlock unlimited', or LimitMEMLOCK=infinity in the systemd unit, then fully restart the service","Or shrink dfs.datanode.max.locked.memory to fit under the current ulimit (the value is bytes; size suffixes like 4gb are accepted)","Verify at runtime: cat /proc/$(pidof DataNode)/limits | grep -i memlock shows the effective ceiling"],"exampleFix":"# before\n# /etc/security/limits.conf: (nothing) -> ulimit -l 65536 bytes\ndfs.datanode.max.locked.memory=4gb\n# after\n# /etc/security/limits.conf\nhdfs soft memlock unlimited\nhdfs hard memlock unlimited\n# systemd unit: LimitMEMLOCK=infinity, then restart the DataNode","handlingStrategy":"validation","validationCode":"long maxLocked = conf.getLongBytes(DFS_DATANODE_MAX_LOCKED_MEMORY_KEY, 0);\nlong memlock = NativeIO.POSIX.getCacheManipulator().getMemlockLimit();\nif (maxLocked > memlock) {\n  throw new IllegalStateException(\"maxLockedMemory \" + maxLocked + \" > RLIMIT_MEMLOCK \" + memlock\n      + \" — raise ulimit -l / LimitMEMLOCK or lower the setting\");\n}","typeGuard":null,"tryCatchPattern":"catch (RuntimeException e) {\n  if (e.getMessage().contains(\"RLIMIT_MEMLOCK\")) {\n    // raise memlock limit for the hdfs service user and restart, or shrink the configured budget\n  }\n}","preventionTips":["Set 'hdfs - memlock unlimited' in limits.conf and LimitMEMLOCK=infinity in systemd units","Check /proc/<pid>/limits after any service-manager change — PAM limits may not apply","Recheck the ulimit whenever dfs.datanode.max.locked.memory is raised"],"tags":["hadoop","hdfs","datanode","ulimit","memlock","centralized-cache","os-limits"],"backgroundTag":"memlock-ulimit-exceeded","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}