{"record":{"id":"74bf073adb10c1d8","repo":"kubernetes/kubernetes","slug":"failed-to-enforce-system-reserved-cgroup-limits-on","errorCode":null,"errorMessage":"Failed to enforce System Reserved Cgroup Limits on %q: %v","messagePattern":"Failed to enforce System Reserved Cgroup Limits on %q: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/kubelet/cm/node_container_manager_linux.go","lineNumber":127,"sourceCode":"\t\t\tfor {\n\t\t\t\terr := cm.cgroupManager.Update(logger, cgroupConfig)\n\t\t\t\tif err == nil {\n\t\t\t\t\tcm.recorder.Event(nodeRef, v1.EventTypeNormal, events.SuccessfulNodeAllocatableEnforcement, \"Updated Node Allocatable limit across pods\")\n\t\t\t\t\treturn\n\t\t\t\t}\n\t\t\t\tmessage := fmt.Sprintf(\"Failed to update Node Allocatable Limits %q: %v\", cm.cgroupRoot, err)\n\t\t\t\tcm.recorder.Event(nodeRef, v1.EventTypeWarning, events.FailedNodeAllocatableEnforcement, message)\n\t\t\t\ttime.Sleep(time.Minute)\n\t\t\t}\n\t\t}()\n\t}\n\t// Now apply kube reserved and system reserved limits if required.\n\tif nc.EnforceNodeAllocatable.Has(kubetypes.SystemReservedEnforcementKey) {\n\t\tlogger.V(2).Info(\"Enforcing system reserved on cgroup\", \"cgroupName\", nc.SystemReservedCgroupName, \"limits\", nc.SystemReserved)\n\t\tif err := cm.enforceExistingCgroup(logger, nc.SystemReservedCgroupName, nc.SystemReserved, false); err != nil {\n\t\t\tmessage := fmt.Sprintf(\"Failed to enforce System Reserved Cgroup Limits on %q: %v\", nc.SystemReservedCgroupName, err)\n\t\t\tcm.recorder.Event(nodeRef, v1.EventTypeWarning, events.FailedNodeAllocatableEnforcement, message)\n\t\t\treturn errors.New(message)\n\t\t}\n\t\tcm.recorder.Eventf(nodeRef, v1.EventTypeNormal, events.SuccessfulNodeAllocatableEnforcement, \"Updated limits on system reserved cgroup %v\", nc.SystemReservedCgroupName)\n\t}\n\tif nc.EnforceNodeAllocatable.Has(kubetypes.KubeReservedEnforcementKey) {\n\t\tlogger.V(2).Info(\"Enforcing kube reserved on cgroup\", \"cgroupName\", nc.KubeReservedCgroupName, \"limits\", nc.KubeReserved)\n\t\tif err := cm.enforceExistingCgroup(logger, nc.KubeReservedCgroupName, nc.KubeReserved, false); err != nil {\n\t\t\tmessage := fmt.Sprintf(\"Failed to enforce Kube Reserved Cgroup Limits on %q: %v\", nc.KubeReservedCgroupName, err)\n\t\t\tcm.recorder.Event(nodeRef, v1.EventTypeWarning, events.FailedNodeAllocatableEnforcement, message)\n\t\t\treturn errors.New(message)\n\t\t}\n\t\tcm.recorder.Eventf(nodeRef, v1.EventTypeNormal, events.SuccessfulNodeAllocatableEnforcement, \"Updated limits on kube reserved cgroup %v\", nc.KubeReservedCgroupName)\n\t}\n\n\tif nc.EnforceNodeAllocatable.Has(kubetypes.SystemReservedCompressibleEnforcementKey) {\n\t\tlogger.V(2).Info(\"Enforcing system reserved compressible on cgroup\", \"cgroupName\", nc.SystemReservedCgroupName, \"limits\", nc.SystemReserved)\n\t\tif err := cm.enforceExistingCgroup(logger, nc.SystemReservedCgroupName, nc.SystemReserved, true); err != nil {\n\t\t\tmessage := fmt.Sprintf(\"Failed to enforce System Reserved Compressible Cgroup Limits on %q: %v\", nc.SystemReservedCgroupName, err)\n\t\t\tcm.recorder.Event(nodeRef, v1.EventTypeWarning, events.FailedNodeAllocatableEnforcement, message)","sourceCodeStart":109,"sourceCodeEnd":145,"githubUrl":"https://github.com/kubernetes/kubernetes/blob/b882c60b4023bdf09264c2d5d30a2cadebc240fb/pkg/kubelet/cm/node_container_manager_linux.go#L109-L145","documentation":"Emitted at node_container_manager_linux.go:122-128 during enforceNodeAllocatableCgroups() when EnforceNodeAllocatable includes SystemReservedEnforcementKey and cm.enforceExistingCgroup() fails to apply System Reserved limits on the system-reserved cgroup. The error is wrapped, a FailedNodeAllocatableEnforcement warning event is recorded, and the function returns the error, blocking kubelet startup.","triggerScenarios":"Kubelet configured with --enforce-node-allocatable=system-reserved and --system-reserved limits; the cgroup (e.g. /system.slice or a custom SystemReservedCgroupName) is missing, read-only, or the update violates current usage.","commonSituations":"The system-reserved cgroup doesn't exist on the node (parent cgroup not created). cgroupfs mounted read-only or in a locked-down mode. Requested system-reserved limits exceed total node capacity or conflict with kube-reserved. Wrong systemd cgroup driver vs cgroupfs driver mismatch. cgroup v1 vs v2 path confusion.","solutions":["Ensure the system-reserved cgroup exists before kubelet starts (create it via systemd slice or mkdir on cgroupfs).","Verify the --cgroup-driver matches the node's actual cgroup driver (systemd vs cgroupfs).","Recheck --system-reserved values are sane relative to node capacity; reduce if too aggressive.","Confirm cgroupfs is mounted rw and the kubelet has CAP_SYS_ADMIN / privileges to write limits."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"// Pre-flight: confirm system-reserved cgroup exists and is writable.\nif !cm.cgroupManager.Exists(systemReservedCgroup) {\n    return fmt.Errorf(\"system-reserved cgroup %q does not exist\", systemReservedCgroup)\n}","typeGuard":null,"tryCatchPattern":"if err := cm.enforceExistingCgroup(logger, nc.SystemReservedCgroupName, nc.SystemReserved, false); err != nil {\n    klog.ErrorS(err, \"failed to enforce system-reserved cgroup limits\", \"cgroup\", nc.SystemReservedCgroupName)\n    // surface actionable guidance and keep kubelet from proceeding\n    return err\n}","preventionTips":["Pre-create the system-reserved cgroup before starting kubelet.","Match --cgroup-driver to the node's actual driver.","Keep --system-reserved values realistic vs node capacity.","Ensure cgroupfs is mounted read-write and kubelet has privileges."],"tags":["kubernetes","kubelet","cgroups","node-allocatable","linux","configuration","go"],"analyzedSha":"b882c60b4023bdf09264c2d5d30a2cadebc240fb","analyzedAt":"2026-08-07T04:07:48.144Z","schemaVersion":2},"datasetVersion":"2026-08-07T07:17:06.508Z"}