{"record":{"id":"f4cb0fc928893443","repo":"k3s-io/k3s","slug":"incorrect-stage-s-found-on-node-s","errorCode":null,"errorMessage":"incorrect stage: %s found on node %s","messagePattern":"incorrect stage: (.+?) found on node (.+?)","errorType":"http","errorClass":null,"httpStatus":500,"severity":"error","filePath":"pkg/server/handlers/secrets-encrypt.go","lineNumber":593,"sourceCode":"\t\t\treturn fmt.Errorf(\"hash does not match between %s and %s\", firstNodeName, node.ObjectMeta.Name)\n\t\t}\n\t}\n\n\tif prevStage == \"\" {\n\t\treturn nil\n\t}\n\n\toldStage, oldHash, err := getEncryptionHashAnnotation(core)\n\tif err != nil {\n\t\treturn err\n\t}\n\n\tencryptionConfigHash, err := secretsencrypt.GenEncryptionConfigHash(runtime)\n\tif err != nil {\n\t\treturn err\n\t}\n\tif !strings.Contains(prevStage, oldStage) {\n\t\treturn fmt.Errorf(\"incorrect stage: %s found on node %s\", oldStage, nodes.Items[0].ObjectMeta.Name)\n\t} else if oldHash != encryptionConfigHash {\n\t\treturn fmt.Errorf(\"invalid hash: %s found on node %s\", oldHash, nodes.Items[0].ObjectMeta.Name)\n\t}\n\n\treturn nil\n}\n","sourceCodeStart":575,"sourceCodeEnd":600,"githubUrl":"https://github.com/k3s-io/k3s/blob/6ba341e396edc16b8dcae978a7c5e3ac7ee5606e/pkg/server/handlers/secrets-encrypt.go#L575-L600","documentation":"verifyEncryptionHashAnnotation checks that the stage recorded on the local node is compatible with the stage transition being requested (prevStage must contain the locally found oldStage). If the cluster's nodes are not on the stage the request assumes, the transition is rejected with the found stage and node name - stages must be executed in order: prepare -> rotate -> ... -> re-encrypt-active.","triggerScenarios":"Requesting 'rotate' or 're-encrypt-active' when nodes still report 'prepare'; skipping a stage; calling a stage twice with the second invocation expecting the completed first one; version/tooling drift where the client and nodes disagree on the current stage.","commonSituations":"Operators jumping straight to re-encrypt after adding a key without completing intermediate stages; a stage that silently failed earlier on some nodes; retrying an interrupted operation out of order.","solutions":["Check actual state first: 'k3s secrets-encrypt status' on every control-plane node.","Run the stages strictly in order (prepare, then rotate per iteration, finally re-encrypt-active) matching what status reports.","If the stage annotation is stale on some nodes, re-run the currently reported stage to completion before advancing.","Do not skip or parallelize stages with scripts; execute them sequentially and verify status between steps."],"exampleFix":"# before\nk3s secrets-encrypt re-encrypt  # nodes still in 'prepare'\n\n# after\nk3s secrets-encrypt rotate       # complete current stage first\nk3s secrets-encrypt re-encrypt","handlingStrategy":"validation","validationCode":"// Read current stage, then only allow the documented successor\ncur := stageFromAnnotation(ann)\nnext := map[string][]string{\"prepare\":{\"rotate\"},\"rotate\":{\"re-encrypt-active\"}}\nif !contains(next[cur], requested) { log.Fatal(\"execute stages in order\") }","typeGuard":"func isValidTransition(from, to string) bool {\n    if from == \"\" { return to == \"prepare\" }\n    switch to {\n    case \"rotate\", \"re-encrypt-active\":\n        return from == \"prepare\" || from == \"rotate\"\n    }\n    return false\n}","tryCatchPattern":null,"preventionTips":["Run 'k3s secrets-encrypt status' before every stage request","Execute stages strictly in order; never skip or parallelize","Verify stage completion on all nodes between steps"],"tags":["secrets-encryption","state-machine","validation"],"backgroundTag":null,"analyzedSha":"6ba341e396edc16b8dcae978a7c5e3ac7ee5606e","analyzedAt":"2026-08-15T16:27:54.286Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}