kubernetes/kubernetes · error

Failed to enforce System Reserved Cgroup Limits on %q: %v

Error message

Failed to enforce System Reserved Cgroup Limits on %q: %v

What it means

Emitted at node_container_manager_linux.go:122-128 during enforceNodeAllocatableCgroups() when EnforceNodeAllocatable includes SystemReservedEnforcementKey and cm.enforceExistingCgroup() fails to apply System Reserved limits on the system-reserved cgroup. The error is wrapped, a FailedNodeAllocatableEnforcement warning event is recorded, and the function returns the error, blocking kubelet startup.

Source

Thrown at pkg/kubelet/cm/node_container_manager_linux.go:127

			for {
				err := cm.cgroupManager.Update(logger, cgroupConfig)
				if err == nil {
					cm.recorder.Event(nodeRef, v1.EventTypeNormal, events.SuccessfulNodeAllocatableEnforcement, "Updated Node Allocatable limit across pods")
					return
				}
				message := fmt.Sprintf("Failed to update Node Allocatable Limits %q: %v", cm.cgroupRoot, err)
				cm.recorder.Event(nodeRef, v1.EventTypeWarning, events.FailedNodeAllocatableEnforcement, message)
				time.Sleep(time.Minute)
			}
		}()
	}
	// Now apply kube reserved and system reserved limits if required.
	if nc.EnforceNodeAllocatable.Has(kubetypes.SystemReservedEnforcementKey) {
		logger.V(2).Info("Enforcing system reserved on cgroup", "cgroupName", nc.SystemReservedCgroupName, "limits", nc.SystemReserved)
		if err := cm.enforceExistingCgroup(logger, nc.SystemReservedCgroupName, nc.SystemReserved, false); err != nil {
			message := fmt.Sprintf("Failed to enforce System Reserved Cgroup Limits on %q: %v", nc.SystemReservedCgroupName, err)
			cm.recorder.Event(nodeRef, v1.EventTypeWarning, events.FailedNodeAllocatableEnforcement, message)
			return errors.New(message)
		}
		cm.recorder.Eventf(nodeRef, v1.EventTypeNormal, events.SuccessfulNodeAllocatableEnforcement, "Updated limits on system reserved cgroup %v", nc.SystemReservedCgroupName)
	}
	if nc.EnforceNodeAllocatable.Has(kubetypes.KubeReservedEnforcementKey) {
		logger.V(2).Info("Enforcing kube reserved on cgroup", "cgroupName", nc.KubeReservedCgroupName, "limits", nc.KubeReserved)
		if err := cm.enforceExistingCgroup(logger, nc.KubeReservedCgroupName, nc.KubeReserved, false); err != nil {
			message := fmt.Sprintf("Failed to enforce Kube Reserved Cgroup Limits on %q: %v", nc.KubeReservedCgroupName, err)
			cm.recorder.Event(nodeRef, v1.EventTypeWarning, events.FailedNodeAllocatableEnforcement, message)
			return errors.New(message)
		}
		cm.recorder.Eventf(nodeRef, v1.EventTypeNormal, events.SuccessfulNodeAllocatableEnforcement, "Updated limits on kube reserved cgroup %v", nc.KubeReservedCgroupName)
	}

	if nc.EnforceNodeAllocatable.Has(kubetypes.SystemReservedCompressibleEnforcementKey) {
		logger.V(2).Info("Enforcing system reserved compressible on cgroup", "cgroupName", nc.SystemReservedCgroupName, "limits", nc.SystemReserved)
		if err := cm.enforceExistingCgroup(logger, nc.SystemReservedCgroupName, nc.SystemReserved, true); err != nil {
			message := fmt.Sprintf("Failed to enforce System Reserved Compressible Cgroup Limits on %q: %v", nc.SystemReservedCgroupName, err)
			cm.recorder.Event(nodeRef, v1.EventTypeWarning, events.FailedNodeAllocatableEnforcement, message)

View on GitHub (pinned to b882c60b40)

Solutions

1. Ensure the system-reserved cgroup exists before kubelet starts (create it via systemd slice or mkdir on cgroupfs).
2. Verify the --cgroup-driver matches the node's actual cgroup driver (systemd vs cgroupfs).
3. Recheck --system-reserved values are sane relative to node capacity; reduce if too aggressive.
4. Confirm cgroupfs is mounted rw and the kubelet has CAP_SYS_ADMIN / privileges to write limits.

// Pre-flight: confirm system-reserved cgroup exists and is writable.
if !cm.cgroupManager.Exists(systemReservedCgroup) {
    return fmt.Errorf("system-reserved cgroup %q does not exist", systemReservedCgroup)
}

if err := cm.enforceExistingCgroup(logger, nc.SystemReservedCgroupName, nc.SystemReserved, false); err != nil {
    klog.ErrorS(err, "failed to enforce system-reserved cgroup limits", "cgroup", nc.SystemReservedCgroupName)
    // surface actionable guidance and keep kubelet from proceeding
    return err
}

When it happens

Trigger: Kubelet configured with --enforce-node-allocatable=system-reserved and --system-reserved limits; the cgroup (e.g. /system.slice or a custom SystemReservedCgroupName) is missing, read-only, or the update violates current usage.

Common situations: The system-reserved cgroup doesn't exist on the node (parent cgroup not created). cgroupfs mounted read-only or in a locked-down mode. Requested system-reserved limits exceed total node capacity or conflict with kube-reserved. Wrong systemd cgroup driver vs cgroupfs driver mismatch. cgroup v1 vs v2 path confusion.

Related errors

• Failed to enforce Kube Reserved Cgroup Limits on %q: %v
• Failed to enforce System Reserved Compressible Cgroup Limits
• Failed to enforce Kube Reserved Compressible Cgroup Limits o
• Cgroup Manager is not supported in this build
• service-account-issuer is a required flag

AI-assisted analysis of kubernetes/kubernetes@b882c60b40 (2026-08-07). Data as JSON: /api/errors/74bf073adb10c1d8. Report an issue: GitHub.