hashicorp/nomad · critical

failed to fetch key from any peer: %v

Error message

failed to fetch key from any peer: %v

What it means

The keyring replication goroutine (keyringReplicator.run) asked every peer via RPC Keyring.Get for a key ID and none returned one (getResp.Key == nil on all attempts). This means the local server cannot obtain the encryption key that exists elsewhere in the cluster, so replication aborts for that key. The error includes the accumulated RPC error for diagnosis.

Source

Thrown at nomad/encrypter.go:1235

		for _, peer := range krr.srv.peersCache.LocalPeers() {
			if peer.Name == self {
				continue
			}

			krr.logger.Trace("attempting to replicate key from peer",
				"id", keyID, "peer", peer.Name)
			err = krr.srv.forwardServer(peer, "Keyring.Get", getReq, getResp)
			if err == nil && getResp.Key != nil {
				break
			}
		}
	}

	if getResp.Key == nil {
		krr.logger.Error("failed to fetch key from any peer",
			"key", keyID, "error", err)
		return fmt.Errorf("failed to fetch key from any peer: %v", err)
	}

	isClusterUpgraded := krr.srv.peersCache.ServersMeetMinimumVersion(
		krr.srv.Region(), minVersionKeyringInRaft, true)

	// In the legacy replication, we toss out the wrapped key because it's
	// always persisted to disk
	_, err = krr.srv.encrypter.AddUnwrappedKey(getResp.Key, isClusterUpgraded)
	if err != nil {
		return fmt.Errorf("failed to add key to keyring: %v", err)
	}

	krr.logger.Debug("added key", "key", keyID)
	return nil
}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Check the logged RPC error to see why peers were unreachable (connection refused, TLS, permission).
2. Ensure at least one server that holds the key is up and reachable from this server.
3. Verify server.address / TLS / gossip settings allow server-to-server RPC between regions.
4. If the cluster was rebuilt, restore the keystore key files from backup onto a server and restart.
5. During rolling upgrades, keep the old servers running until the new server's log shows the key replicated.

Example fix

// before: all key-holding servers removed
nomad server -join=newservers  # old servers with keystore stopped first
// after
nomad server force-leave old-node  # only AFTER new servers confirm key replication

// before relying on replication, check peer reachability
for _, peer := range peers {
	if _, err := rpcKeyringGet(peer, keyID); err != nil {
		log.Printf("peer %s unreachable: %v", peer, err)
	}
}

for attempt := 0; attempt < 5; attempt++ {
	if err := replicateKey(); err == nil {
		break
	} else if strings.Contains(err.Error(), "failed to fetch key from any peer") {
		time.Sleep(backoff(attempt))
	}
}

When it happens

Trigger: The replicator observes a key ID in raft/peer state that no reachable server can serve: all peers with the key are down, the RPC to each peer fails (network, TLS, ACL), or the key was garbage-collected on the serving peers.

Common situations: Rolling upgrade where old servers were removed before the new one replicated keys; network partition between datacenters/regions; TLS or gossip misconfiguration blocking Keyring.Get RPCs; key rotated and deleted on peers before replication completed.

Related errors

• failed to add key to keyring: %v
• only one server.keyring can be active in Nomad Community Edi
• Failed to start RPC layer: %v
• must pass non-nil job
• job is missing ID

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/87a5f196cabeef25. Report an issue: GitHub.