{"record":{"id":"c1c760cd823b15c1","repo":"hashicorp/nomad","slug":"failed-to-add-key-to-keyring-v","errorCode":null,"errorMessage":"failed to add key to keyring: %v","messagePattern":"failed to add key to keyring: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"nomad/encrypter.go","lineNumber":1245,"sourceCode":"\t\t\t\tbreak\n\t\t\t}\n\t\t}\n\t}\n\n\tif getResp.Key == nil {\n\t\tkrr.logger.Error(\"failed to fetch key from any peer\",\n\t\t\t\"key\", keyID, \"error\", err)\n\t\treturn fmt.Errorf(\"failed to fetch key from any peer: %v\", err)\n\t}\n\n\tisClusterUpgraded := krr.srv.peersCache.ServersMeetMinimumVersion(\n\t\tkrr.srv.Region(), minVersionKeyringInRaft, true)\n\n\t// In the legacy replication, we toss out the wrapped key because it's\n\t// always persisted to disk\n\t_, err = krr.srv.encrypter.AddUnwrappedKey(getResp.Key, isClusterUpgraded)\n\tif err != nil {\n\t\treturn fmt.Errorf(\"failed to add key to keyring: %v\", err)\n\t}\n\n\tkrr.logger.Debug(\"added key\", \"key\", keyID)\n\treturn nil\n}\n","sourceCodeStart":1227,"sourceCodeEnd":1251,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/encrypter.go#L1227-L1251","documentation":"After successfully fetching a key from a peer, the replicator calls encrypter.AddUnwrappedKey to add it to the local keyring, and that call failed. The error is a wrapper; the interesting cause is inside AddUnwrappedKey (e.g. cipher configuration failure or keyring insertion failure). Replication for this key stops and will be retried by run().","triggerScenarios":"replicateKey gets a valid key from a peer but AddUnwrappedKey rejects it: bad key length/type for the AEAD cipher, the key already exists in an incompatible form, or the local keyring is in a bad state after a downgrade.","commonSituations":"Version skew: cluster upgraded past minVersionKeyringInRaft with mismatched key formats; corrupted in-memory keyring after a partial restore; key material from peers failing local cipher validation (same family as the 'could not configure cipher' error).","solutions":["Check the inner error from AddUnwrappedKey for the actual cause.","Verify all servers run compatible Nomad versions (peersCache.ServersMeetMinimumVersion) before mixing key formats.","Confirm the fetched key bytes are intact — check for corruption between peers (TLS, disk).","If keyring state is corrupt, restore the data/keystore files from a healthy server backup and restart.","Restart the server so the keyring reinitializes and replication retries cleanly."],"exampleFix":"// before: server on old version joins upgraded cluster\nversion = 1.5.x  // below minVersionKeyringInRaft\n// after\nupgrade server binary to match cluster >= minVersionKeyringInRaft, then restart","handlingStrategy":"validation","validationCode":"if !peersCache.ServersMeetMinimumVersion(region, minVersionKeyringInRaft, true) {\n\treturn fmt.Errorf(\"cluster below min version for raft keyring; upgrade all servers first\")\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Keep all servers on the same Nomad version before key rotation","Log the inner error from AddUnwrappedKey, not just the wrapper","Snapshot the keystore directory before version upgrades"],"tags":["go","nomad","keyring","replication","encryption"],"backgroundTag":"invalid-key-material","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}