{"record":{"id":"209b27c8da7af50e","repo":"k3s-io/k3s","slug":"node-s-is-running-k3s-version-s-that-does-not-su","errorCode":null,"errorMessage":"node %s is running k3s version %s that does not support rotate-keys","messagePattern":"node (.+?) is running k3s version (.+?) that does not support rotate-keys","errorType":"http","errorClass":null,"httpStatus":400,"severity":"error","filePath":"pkg/server/handlers/secrets-encrypt.go","lineNumber":551,"sourceCode":"\n// verifyRotateKeysSupport checks that the k3s version is at least v1.28.0 on all control-plane nodes\nfunc verifyRotateKeysSupport(core core.Interface) error {\n\tlabelSelector := labels.Set{util.ControlPlaneRoleLabelKey: \"true\"}.String()\n\tnodes, err := core.V1().Node().List(metav1.ListOptions{LabelSelector: labelSelector})\n\tif err != nil {\n\t\treturn err\n\t}\n\tfor _, node := range nodes.Items {\n\t\tkubver, err := semver.ParseTolerant(node.Status.NodeInfo.KubeletVersion)\n\t\tif err != nil {\n\t\t\treturn fmt.Errorf(\"failed to parse kubelet version %s: %v\", node.Status.NodeInfo.KubeletVersion, err)\n\t\t}\n\t\tsupportVer, err := semver.Make(\"1.28.0\")\n\t\tif err != nil {\n\t\t\treturn err\n\t\t}\n\t\tif kubver.LT(supportVer) {\n\t\t\treturn fmt.Errorf(\"node %s is running k3s version %s that does not support rotate-keys\", node.ObjectMeta.Name, kubver.String())\n\t\t}\n\t}\n\treturn nil\n}\n\n// verifyEncryptionHashAnnotation checks that all nodes are on the same stage,\n// and that a request for new stage is valid\nfunc verifyEncryptionHashAnnotation(runtime *config.ControlRuntime, core core.Interface, prevStage string) error {\n\tvar firstHash string\n\tvar firstNodeName string\n\tfirst := true\n\tlabelSelector := labels.Set{util.ControlPlaneRoleLabelKey: \"true\"}.String()\n\tnodes, err := core.V1().Node().List(metav1.ListOptions{LabelSelector: labelSelector})\n\tif err != nil {\n\t\treturn err\n\t}\n\tfor _, node := range nodes.Items {\n\t\thash, ok := node.Annotations[secretsencrypt.EncryptionHashAnnotation]","sourceCodeStart":533,"sourceCodeEnd":569,"githubUrl":"https://github.com/k3s-io/k3s/blob/6ba341e396edc16b8dcae978a7c5e3ac7ee5606e/pkg/server/handlers/secrets-encrypt.go#L533-L569","documentation":"Key rotation without downtime requires every control-plane node to be able to read both old and new key sets, a capability introduced in k3s/Kubernetes 1.28. verifyRotateKeysSupport compares each node's parsed kubelet version against 1.28.0 and rejects rotate-keys when any control-plane node is older.","triggerScenarios":"'k3s secrets-encrypt rotate-keys' on a mixed-version control plane where at least one node reports KubeletVersion < 1.28.0 - typically mid-upgrade, one server not yet upgraded, or an old server left behind.","commonSituations":"Rolling upgrade in progress when the operator runs rotate-keys; a forgotten etcd/control-plane node still on 1.27; nodes upgraded out of order violating the version-skew policy.","solutions":["Finish upgrading every control-plane node to at least v1.28: kubectl get nodes -L node-role.kubernetes.io/control-plane and check versions.","Retry 'k3s secrets-encrypt rotate-keys' once all servers report >= 1.28.","If an old node cannot be upgraded, remove it from the control plane before rotating keys.","Sequence encryption operations after upgrades, not during them."],"exampleFix":"# before\nn1 v1.28.4+k3s3 (control-plane)\nn2 v1.27.13+k3s1 (control-plane) -> k3s secrets-encrypt rotate-keys fails\n\n# after: upgrade n2, then\nk3s secrets-encrypt rotate-keys  # succeeds","handlingStrategy":"validation","validationCode":"// Require >= 1.28.0 on every control-plane node before rotate-keys\nmin, _ := semver.Make(\"1.28.0\")\nfor _, n := range nodes.Items {\n    v, err := semver.ParseTolerant(n.Status.NodeInfo.KubeletVersion)\n    if err != nil || v.LT(min) {\n        log.Fatalf(\"node %s below 1.28 - upgrade before rotate-keys\", n.Name)\n    }\n}","typeGuard":"func supportsRotateKeys(v string) bool {\n    ver, err := semver.ParseTolerant(v)\n    min, _ := semver.Make(\"1.28.0\")\n    return err == nil && !ver.LT(min)\n}","tryCatchPattern":null,"preventionTips":["Finish control-plane upgrades before rotating keys","Check kubectl get nodes output for stragglers","Remove unupgradable control-plane nodes from rotation scope first"],"tags":["upgrade","versioning","secrets-encryption"],"backgroundTag":null,"analyzedSha":"6ba341e396edc16b8dcae978a7c5e3ac7ee5606e","analyzedAt":"2026-08-15T16:27:54.286Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}