{"record":{"id":"87a5f196cabeef25","repo":"hashicorp/nomad","slug":"failed-to-fetch-key-from-any-peer-v","errorCode":null,"errorMessage":"failed to fetch key from any peer: %v","messagePattern":"failed to fetch key from any peer: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"nomad/encrypter.go","lineNumber":1235,"sourceCode":"\n\t\tfor _, peer := range krr.srv.peersCache.LocalPeers() {\n\t\t\tif peer.Name == self {\n\t\t\t\tcontinue\n\t\t\t}\n\n\t\t\tkrr.logger.Trace(\"attempting to replicate key from peer\",\n\t\t\t\t\"id\", keyID, \"peer\", peer.Name)\n\t\t\terr = krr.srv.forwardServer(peer, \"Keyring.Get\", getReq, getResp)\n\t\t\tif err == nil && getResp.Key != nil {\n\t\t\t\tbreak\n\t\t\t}\n\t\t}\n\t}\n\n\tif getResp.Key == nil {\n\t\tkrr.logger.Error(\"failed to fetch key from any peer\",\n\t\t\t\"key\", keyID, \"error\", err)\n\t\treturn fmt.Errorf(\"failed to fetch key from any peer: %v\", err)\n\t}\n\n\tisClusterUpgraded := krr.srv.peersCache.ServersMeetMinimumVersion(\n\t\tkrr.srv.Region(), minVersionKeyringInRaft, true)\n\n\t// In the legacy replication, we toss out the wrapped key because it's\n\t// always persisted to disk\n\t_, err = krr.srv.encrypter.AddUnwrappedKey(getResp.Key, isClusterUpgraded)\n\tif err != nil {\n\t\treturn fmt.Errorf(\"failed to add key to keyring: %v\", err)\n\t}\n\n\tkrr.logger.Debug(\"added key\", \"key\", keyID)\n\treturn nil\n}\n","sourceCodeStart":1217,"sourceCodeEnd":1251,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/encrypter.go#L1217-L1251","documentation":"The keyring replication goroutine (keyringReplicator.run) asked every peer via RPC Keyring.Get for a key ID and none returned one (getResp.Key == nil on all attempts). This means the local server cannot obtain the encryption key that exists elsewhere in the cluster, so replication aborts for that key. The error includes the accumulated RPC error for diagnosis.","triggerScenarios":"The replicator observes a key ID in raft/peer state that no reachable server can serve: all peers with the key are down, the RPC to each peer fails (network, TLS, ACL), or the key was garbage-collected on the serving peers.","commonSituations":"Rolling upgrade where old servers were removed before the new one replicated keys; network partition between datacenters/regions; TLS or gossip misconfiguration blocking Keyring.Get RPCs; key rotated and deleted on peers before replication completed.","solutions":["Check the logged RPC error to see why peers were unreachable (connection refused, TLS, permission).","Ensure at least one server that holds the key is up and reachable from this server.","Verify server.address / TLS / gossip settings allow server-to-server RPC between regions.","If the cluster was rebuilt, restore the keystore key files from backup onto a server and restart.","During rolling upgrades, keep the old servers running until the new server's log shows the key replicated."],"exampleFix":"// before: all key-holding servers removed\nnomad server -join=newservers  # old servers with keystore stopped first\n// after\nnomad server force-leave old-node  # only AFTER new servers confirm key replication","handlingStrategy":"retry","validationCode":"// before relying on replication, check peer reachability\nfor _, peer := range peers {\n\tif _, err := rpcKeyringGet(peer, keyID); err != nil {\n\t\tlog.Printf(\"peer %s unreachable: %v\", peer, err)\n\t}\n}","typeGuard":null,"tryCatchPattern":"for attempt := 0; attempt < 5; attempt++ {\n\tif err := replicateKey(); err == nil {\n\t\tbreak\n\t} else if strings.Contains(err.Error(), \"failed to fetch key from any peer\") {\n\t\ttime.Sleep(backoff(attempt))\n\t}\n}","preventionTips":["Keep at least one key-holding server up during rolling upgrades","Monitor server-to-server RPC health (TLS, ports, gossip)","Back up data/keystore across servers","Delay force-leave/retire of old servers until keys are replicated"],"tags":["go","nomad","keyring","replication","rpc"],"backgroundTag":"peer-unavailable","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}