{"record":{"id":"3ce607cce9ceae74","repo":"k3s-io/k3s","slug":"invalid-annotation-s-found-on-node-s","errorCode":null,"errorMessage":"invalid annotation %s found on node %s","messagePattern":"invalid annotation (.+?) found on node (.+?)","errorType":"http","errorClass":null,"httpStatus":400,"severity":"error","filePath":"pkg/server/handlers/secrets-encrypt.go","lineNumber":527,"sourceCode":"\t\tkeys.SBKeys = append(keys.SBKeys, newKey...)\n\t}\n\tlogrus.Infoln(\"Adding secrets-encryption key: \", newKey)\n\treturn nil\n}\n\nfunc getEncryptionHashAnnotation(core core.Interface) (string, string, error) {\n\tnodeName := os.Getenv(\"NODE_NAME\")\n\tnode, err := core.V1().Node().Get(nodeName, metav1.GetOptions{})\n\tif err != nil {\n\t\treturn \"\", \"\", err\n\t}\n\tif _, ok := node.Labels[util.ControlPlaneRoleLabelKey]; !ok {\n\t\treturn \"\", \"\", fmt.Errorf(\"cannot manage secrets encryption on non control-plane node %s\", nodeName)\n\t}\n\tif ann, ok := node.Annotations[secretsencrypt.EncryptionHashAnnotation]; ok {\n\t\tsplit := strings.Split(ann, \"-\")\n\t\tif len(split) != 2 {\n\t\t\treturn \"\", \"\", fmt.Errorf(\"invalid annotation %s found on node %s\", ann, nodeName)\n\t\t}\n\t\treturn split[0], split[1], nil\n\t}\n\treturn \"\", \"\", fmt.Errorf(\"missing annotation on node %s\", nodeName)\n}\n\n// verifyRotateKeysSupport checks that the k3s version is at least v1.28.0 on all control-plane nodes\nfunc verifyRotateKeysSupport(core core.Interface) error {\n\tlabelSelector := labels.Set{util.ControlPlaneRoleLabelKey: \"true\"}.String()\n\tnodes, err := core.V1().Node().List(metav1.ListOptions{LabelSelector: labelSelector})\n\tif err != nil {\n\t\treturn err\n\t}\n\tfor _, node := range nodes.Items {\n\t\tkubver, err := semver.ParseTolerant(node.Status.NodeInfo.KubeletVersion)\n\t\tif err != nil {\n\t\t\treturn fmt.Errorf(\"failed to parse kubelet version %s: %v\", node.Status.NodeInfo.KubeletVersion, err)\n\t\t}","sourceCodeStart":509,"sourceCodeEnd":545,"githubUrl":"https://github.com/k3s-io/k3s/blob/6ba341e396edc16b8dcae978a7c5e3ac7ee5606e/pkg/server/handlers/secrets-encrypt.go#L509-L545","documentation":"The node annotation used to coordinate secrets-encryption stages stores '<sha256-hash>-<stage>' (split on '-'). getEncryptionHashAnnotation validates that the value splits into exactly two parts; a value with zero, or two or more dashes fails this check and names the offending node.","triggerScenarios":"The EncryptionHashAnnotation on the local control-plane node contains a malformed value: manually edited to plain text, a hash with the stage appended twice, or pasted with stray characters. getEncryptionHashAnnotation splits on '-' and len(split) != 2 triggers the error.","commonSituations":"Operators copying annotation values between nodes with kubectl annotate and truncating/duplicating part; external automation writing its own annotation format; leftovers from a failed or interrupted manual migration.","solutions":["Read the current value: kubectl get node <name> -o jsonpath='{.metadata.annotations}' and inspect the encryption-hash annotation.","Copy the exact intact '<hash>-<stage>' value from a healthy control-plane node running the same stage, or delete the annotation cluster-wide and re-run 'secrets-encrypt prepare' to regenerate consistent state.","Never hand-edit the annotation; drive all stage transitions through the k3s CLI so the writer keeps the format.","Re-check 'k3s secrets-encrypt status' on all servers after repairing."],"exampleFix":"# before\nkubectl annotate node n1 encryption.hash=\"stage-prepare-abc123\" --overwrite  # 3 parts\n\n# after\nkubectl annotate node n1 encryption.hash=\"abc123-prepare\" --overwrite  # hash-stage","handlingStrategy":"validation","validationCode":"// Validate annotation shape before invoking stages\nparts := strings.Split(ann, \"-\")\nif len(parts) != 2 || parts[0] == \"\" || parts[1] == \"\" {\n    log.Fatalf(\"malformed encryption hash annotation %q - repair before continuing\", ann)\n}","typeGuard":"func validHashAnnotation(ann string) bool {\n    p := strings.Split(ann, \"-\")\n    return len(p) == 2 && p[0] != \"\" && p[1] != \"\"\n}","tryCatchPattern":null,"preventionTips":["Never hand-edit the encryption-hash annotation","Copy annotations verbatim (no truncation) when migrating","Drive stage transitions exclusively via the CLI"],"tags":["annotations","secrets-encryption","cluster-sync"],"backgroundTag":null,"analyzedSha":"6ba341e396edc16b8dcae978a7c5e3ac7ee5606e","analyzedAt":"2026-08-15T16:27:54.286Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}