k3s-io/k3s · error

node %s is running k3s version %s that does not support rota

Error message

node %s is running k3s version %s that does not support rotate-keys

What it means

Key rotation without downtime requires every control-plane node to be able to read both old and new key sets, a capability introduced in k3s/Kubernetes 1.28. verifyRotateKeysSupport compares each node's parsed kubelet version against 1.28.0 and rejects rotate-keys when any control-plane node is older.

Source

Thrown at pkg/server/handlers/secrets-encrypt.go:551

// verifyRotateKeysSupport checks that the k3s version is at least v1.28.0 on all control-plane nodes
func verifyRotateKeysSupport(core core.Interface) error {
	labelSelector := labels.Set{util.ControlPlaneRoleLabelKey: "true"}.String()
	nodes, err := core.V1().Node().List(metav1.ListOptions{LabelSelector: labelSelector})
	if err != nil {
		return err
	}
	for _, node := range nodes.Items {
		kubver, err := semver.ParseTolerant(node.Status.NodeInfo.KubeletVersion)
		if err != nil {
			return fmt.Errorf("failed to parse kubelet version %s: %v", node.Status.NodeInfo.KubeletVersion, err)
		}
		supportVer, err := semver.Make("1.28.0")
		if err != nil {
			return err
		}
		if kubver.LT(supportVer) {
			return fmt.Errorf("node %s is running k3s version %s that does not support rotate-keys", node.ObjectMeta.Name, kubver.String())
		}
	}
	return nil
}

// verifyEncryptionHashAnnotation checks that all nodes are on the same stage,
// and that a request for new stage is valid
func verifyEncryptionHashAnnotation(runtime *config.ControlRuntime, core core.Interface, prevStage string) error {
	var firstHash string
	var firstNodeName string
	first := true
	labelSelector := labels.Set{util.ControlPlaneRoleLabelKey: "true"}.String()
	nodes, err := core.V1().Node().List(metav1.ListOptions{LabelSelector: labelSelector})
	if err != nil {
		return err
	}
	for _, node := range nodes.Items {
		hash, ok := node.Annotations[secretsencrypt.EncryptionHashAnnotation]

View on GitHub (pinned to 6ba341e396)

Solutions

1. Finish upgrading every control-plane node to at least v1.28: kubectl get nodes -L node-role.kubernetes.io/control-plane and check versions.
2. Retry 'k3s secrets-encrypt rotate-keys' once all servers report >= 1.28.
3. If an old node cannot be upgraded, remove it from the control plane before rotating keys.
4. Sequence encryption operations after upgrades, not during them.

Example fix

# before
n1 v1.28.4+k3s3 (control-plane)
n2 v1.27.13+k3s1 (control-plane) -> k3s secrets-encrypt rotate-keys fails

# after: upgrade n2, then
k3s secrets-encrypt rotate-keys  # succeeds

// Require >= 1.28.0 on every control-plane node before rotate-keys
min, _ := semver.Make("1.28.0")
for _, n := range nodes.Items {
    v, err := semver.ParseTolerant(n.Status.NodeInfo.KubeletVersion)
    if err != nil || v.LT(min) {
        log.Fatalf("node %s below 1.28 - upgrade before rotate-keys", n.Name)
    }
}

func supportsRotateKeys(v string) bool {
    ver, err := semver.ParseTolerant(v)
    min, _ := semver.Make("1.28.0")
    return err == nil && !ver.LT(min)
}

When it happens

Trigger: 'k3s secrets-encrypt rotate-keys' on a mixed-version control plane where at least one node reports KubeletVersion < 1.28.0 - typically mid-upgrade, one server not yet upgraded, or an old server left behind.

Common situations: Rolling upgrade in progress when the operator runs rotate-keys; a forgotten etcd/control-plane node still on 1.27; nodes upgraded out of order violating the version-skew policy.

Related errors

• incompatible down-level server detected; servers must be upg
• failed to parse kubelet version %s: %v
• unable to enable/disable secrets encryption, unknown configu
• method not allowed
• prepare does not support secretbox key type, use rotate-keys

AI-assisted analysis of k3s-io/k3s@6ba341e396 (2026-08-15). Data as JSON: /api/errors/209b27c8da7af50e. Report an issue: GitHub.