{"record":{"id":"ffacec58beaa1575","repo":"hashicorp/nomad","slug":"rotated-key-does-not-exist-in-keyring-w","errorCode":null,"errorMessage":"rotated key does not exist in keyring: %w","messagePattern":"rotated key does not exist in keyring: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/core_sched.go","lineNumber":1268,"sourceCode":"\t\t\treturn err\n\t\t}\n\n\t\t// Perform the re-encryption of variables using the new active key. If\n\t\t// we reach a timeout, there is no need to return an error, as a new\n\t\t// eval will be emitted to continue the work. We do not mark the key\n\t\t// as inactive until all variables have been rekeyed. If any other error\n\t\t// occurs, we return it to the caller.\n\t\tif err = c.rotateVariables(varIter, eval); err != nil {\n\t\t\tif errors.Is(err, context.DeadlineExceeded) {\n\t\t\t\tc.logger.Info(\"timeout reached rekeying variables\", \"key_id\", wrappedKeys.KeyID)\n\t\t\t\treturn nil\n\t\t\t}\n\t\t\treturn err\n\t\t}\n\n\t\trootKey, err := c.srv.encrypter.GetKey(wrappedKeys.KeyID)\n\t\tif err != nil {\n\t\t\treturn fmt.Errorf(\"rotated key does not exist in keyring: %w\", err)\n\t\t}\n\t\trootKey = rootKey.MakeInactive()\n\n\t\treq := &structs.KeyringUpdateRootKeyRequest{\n\t\t\tRootKey: rootKey,\n\t\t\tWriteRequest: structs.WriteRequest{\n\t\t\t\tRegion:    c.srv.config.Region,\n\t\t\t\tAuthToken: eval.LeaderACL,\n\t\t\t},\n\t\t}\n\t\tif err := c.srv.RPC(\"Keyring.Update\",\n\t\t\treq, &structs.KeyringUpdateRootKeyResponse{}); err != nil {\n\t\t\tc.logger.Error(\"rekey complete but failed to mark key as inactive\", \"error\", err)\n\t\t\treturn err\n\t\t}\n\n\t\t// Log a success, so cluster operators can see that the rekey has\n\t\t// completed successfully.","sourceCodeStart":1250,"sourceCodeEnd":1286,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/core_sched.go#L1250-L1286","documentation":"During a keyring rotation (variablesRekey), the server wraps data with the new root key and then looks the key up via the encrypter. If GetKey returns an error, the rotated key ID is not present in the local keyring, meaning the cluster's keyrings are out of sync or the key was removed before rotation completed.","triggerScenarios":"Calling the keyring rotation path when wrappedKeys.KeyID refers to a root key absent from the local keyring store, e.g. the key was deleted or the keyring replication lagged behind the leader that generated it.","commonSituations":"Running rotation on a server that has not yet replicated the new key; an operator concurrently removed the key from the keyring; restoring a server from a snapshot missing newer keys.","solutions":["Verify the key exists with nomad operator root keyring list and re-run rotation after replication completes","Check keyring replication/network connectivity between servers","Restore the missing key into the keyring before retrying rotation"],"exampleFix":null,"handlingStrategy":"validation","validationCode":"// precheck: ensure the key exists before rotation\nif _, err := encrypter.GetKey(keyID); err != nil {\n    return fmt.Errorf(\"skip rotation, key %s not in keyring\", keyID)\n}","typeGuard":null,"tryCatchPattern":"// wrap and inspect\nif err := rotateKeys(); err != nil {\n    if strings.Contains(err.Error(), \"does not exist in keyring\") {\n        // resync/retry after replication\n    }\n}","preventionTips":["List keyring keys before rotating to confirm the target key exists","Allow keyring replication to settle across servers before rotation","Avoid deleting root keys while rotation jobs are pending"],"tags":["keyring","encryption","state-consistency"],"backgroundTag":"keyring-key-not-found","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}