{"record":{"id":"c2d55a3cacbbbd57","repo":"k3s-io/k3s","slug":"invalid-hash-s-found-on-node-s","errorCode":null,"errorMessage":"invalid hash: %s found on node %s","messagePattern":"invalid hash: (.+?) found on node (.+?)","errorType":"http","errorClass":null,"httpStatus":500,"severity":"error","filePath":"pkg/server/handlers/secrets-encrypt.go","lineNumber":595,"sourceCode":"\t}\n\n\tif prevStage == \"\" {\n\t\treturn nil\n\t}\n\n\toldStage, oldHash, err := getEncryptionHashAnnotation(core)\n\tif err != nil {\n\t\treturn err\n\t}\n\n\tencryptionConfigHash, err := secretsencrypt.GenEncryptionConfigHash(runtime)\n\tif err != nil {\n\t\treturn err\n\t}\n\tif !strings.Contains(prevStage, oldStage) {\n\t\treturn fmt.Errorf(\"incorrect stage: %s found on node %s\", oldStage, nodes.Items[0].ObjectMeta.Name)\n\t} else if oldHash != encryptionConfigHash {\n\t\treturn fmt.Errorf(\"invalid hash: %s found on node %s\", oldHash, nodes.Items[0].ObjectMeta.Name)\n\t}\n\n\treturn nil\n}\n","sourceCodeStart":577,"sourceCodeEnd":600,"githubUrl":"https://github.com/k3s-io/k3s/blob/6ba341e396edc16b8dcae978a7c5e3ac7ee5606e/pkg/server/handlers/secrets-encrypt.go#L577-L600","documentation":"After stage agreement, verifyEncryptionHashAnnotation recomputes the sha256 of the local encryption configuration (GenEncryptionConfigHash) and compares it with the hash stored in the node annotation. A mismatch means the on-disk encryption-config.yaml is not the file the annotation was written for - the config drifted from the recorded state.","triggerScenarios":"The annotation survived but /var/lib/rancher/k3s/server/cred/encryption-config.yaml was modified, replaced with a version from another node/backup, or truncated/restored incompletely; any stage request then fails the oldHash != encryptionConfigHash check on nodes.Items[0].","commonSituations":"Restoring server state from backup that pairs a new config with an old annotation (or vice versa); hand-editing key order or formatting in the YAML (hash is over exact bytes); copying cred files between nodes with mismatched annotations.","solutions":["Make config and annotation consistent again: copy the exact encryption-config.yaml (and keys) from a control-plane node whose 'secrets-encrypt status' is healthy, then let the annotation be rewritten by re-running the current stage.","If no good copy exists cluster-wide, export secrets safely, remove the encryption config, and re-bootstrap encryption via enable + prepare.","Never edit the YAML by hand or restore the cred directory partially; restore data-dir snapshots atomically.","Verify with 'k3s secrets-encrypt status' that hash and stage agree on all servers before advancing."],"exampleFix":"# restore exact config from the healthy node, then re-sync\nscp healthy:/var/lib/rancher/k3s/server/cred/encryption-config.yaml /var/lib/rancher/k3s/server/cred/\nsystemctl restart k3s && k3s secrets-encrypt status","handlingStrategy":"validation","validationCode":"// Recompute config hash and compare with annotation before stage calls\nh, err := secretsencrypt.GenEncryptionConfigHash(runtime)\nif err != nil { log.Fatal(err) }\nif h != annHash { log.Fatal(\"on-disk config does not match recorded hash - restore consistent state\") }","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Never hand-edit encryption-config.yaml (hash covers exact bytes)","Restore data-dir snapshots atomically so config and annotations stay paired","Copy full cred directories between nodes, not single files"],"tags":["secrets-encryption","integrity","configuration"],"backgroundTag":null,"analyzedSha":"6ba341e396edc16b8dcae978a7c5e3ac7ee5606e","analyzedAt":"2026-08-15T16:27:54.286Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}