{"record":{"id":"0225c9066d57aae7","repo":"k3s-io/k3s","slug":"hash-does-not-match-between-s-and-s","errorCode":null,"errorMessage":"hash does not match between %s and %s","messagePattern":"hash does not match between (.+?) and (.+?)","errorType":"http","errorClass":null,"httpStatus":400,"severity":"error","filePath":"pkg/server/handlers/secrets-encrypt.go","lineNumber":575,"sourceCode":"// verifyEncryptionHashAnnotation checks that all nodes are on the same stage,\n// and that a request for new stage is valid\nfunc verifyEncryptionHashAnnotation(runtime *config.ControlRuntime, core core.Interface, prevStage string) error {\n\tvar firstHash string\n\tvar firstNodeName string\n\tfirst := true\n\tlabelSelector := labels.Set{util.ControlPlaneRoleLabelKey: \"true\"}.String()\n\tnodes, err := core.V1().Node().List(metav1.ListOptions{LabelSelector: labelSelector})\n\tif err != nil {\n\t\treturn err\n\t}\n\tfor _, node := range nodes.Items {\n\t\thash, ok := node.Annotations[secretsencrypt.EncryptionHashAnnotation]\n\t\tif ok && first {\n\t\t\tfirstHash = hash\n\t\t\tfirst = false\n\t\t\tfirstNodeName = node.ObjectMeta.Name\n\t\t} else if ok && hash != firstHash {\n\t\t\treturn fmt.Errorf(\"hash does not match between %s and %s\", firstNodeName, node.ObjectMeta.Name)\n\t\t}\n\t}\n\n\tif prevStage == \"\" {\n\t\treturn nil\n\t}\n\n\toldStage, oldHash, err := getEncryptionHashAnnotation(core)\n\tif err != nil {\n\t\treturn err\n\t}\n\n\tencryptionConfigHash, err := secretsencrypt.GenEncryptionConfigHash(runtime)\n\tif err != nil {\n\t\treturn err\n\t}\n\tif !strings.Contains(prevStage, oldStage) {\n\t\treturn fmt.Errorf(\"incorrect stage: %s found on node %s\", oldStage, nodes.Items[0].ObjectMeta.Name)","sourceCodeStart":557,"sourceCodeEnd":593,"githubUrl":"https://github.com/k3s-io/k3s/blob/6ba341e396edc16b8dcae978a7c5e3ac7ee5606e/pkg/server/handlers/secrets-encrypt.go#L557-L593","documentation":"verifyEncryptionHashAnnotation lists all control-plane nodes and requires that every node carrying the encryption hash annotation reports the identical value (same config hash and stage). The first annotated node becomes the baseline; any other annotated node with a different value aborts stage verification with this error naming both nodes.","triggerScenarios":"A secrets-encrypt stage request when two or more control-plane nodes hold different '<hash>-<stage>' annotations: one server was down while a stage advanced, an encryption config was restored on one node from backup, or a node rejoined with stale state.","commonSituations":"A powered-off server during 'secrets-encrypt prepare/rotate'; restoring /var/lib/rancher/k3s/server/cred on one node from an older snapshot; partial stage application interrupted mid-rollout.","solutions":["Identify the divergent nodes: kubectl get nodes -o custom-columns=NAME:.metadata.name,ANN:.metadata.annotations -l node-role.kubernetes.io/control-plane.","Converge state: either restore that node's encryption-config.yaml and keys from the majority/current node, or let it re-sync by restarting k3s on it after copying cred files.","Re-run the same stage so all nodes rewrite a consistent annotation.","Avoid powering down servers across an encryption stage boundary; complete stages before maintenance."],"exampleFix":"# converge out-of-sync node (run on the stale node)\nsystemctl stop k3s\nscp current:/var/lib/rancher/k3s/server/cred/encryption-config.yaml /var/lib/rancher/k3s/server/cred/\nsystemctl start k3s && k3s secrets-encrypt status","handlingStrategy":"validation","validationCode":"// Verify all annotated control-plane nodes agree before stage requests\nvar first string\nfor _, n := range nodes.Items {\n    if h, ok := n.Annotations[\"encryption.hash\"]; ok {\n        if first == \"\" { first = h } else if h != first {\n            log.Fatalf(\"annotation divergence between nodes - converge first\")\n        }\n    }\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Keep all servers running during stage transitions","Restore cred directories atomically (config + keys + annotation together)","Compare 'k3s secrets-encrypt status' across servers before advancing stages"],"tags":["secrets-encryption","cluster-sync","annotations"],"backgroundTag":null,"analyzedSha":"6ba341e396edc16b8dcae978a7c5e3ac7ee5606e","analyzedAt":"2026-08-15T16:27:54.286Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}