hashicorp/nomad · critical

failed to add key to keyring: %v

Error message

failed to add key to keyring: %v

What it means

After successfully fetching a key from a peer, the replicator calls encrypter.AddUnwrappedKey to add it to the local keyring, and that call failed. The error is a wrapper; the interesting cause is inside AddUnwrappedKey (e.g. cipher configuration failure or keyring insertion failure). Replication for this key stops and will be retried by run().

Source

Thrown at nomad/encrypter.go:1245

				break
			}
		}
	}

	if getResp.Key == nil {
		krr.logger.Error("failed to fetch key from any peer",
			"key", keyID, "error", err)
		return fmt.Errorf("failed to fetch key from any peer: %v", err)
	}

	isClusterUpgraded := krr.srv.peersCache.ServersMeetMinimumVersion(
		krr.srv.Region(), minVersionKeyringInRaft, true)

	// In the legacy replication, we toss out the wrapped key because it's
	// always persisted to disk
	_, err = krr.srv.encrypter.AddUnwrappedKey(getResp.Key, isClusterUpgraded)
	if err != nil {
		return fmt.Errorf("failed to add key to keyring: %v", err)
	}

	krr.logger.Debug("added key", "key", keyID)
	return nil
}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Check the inner error from AddUnwrappedKey for the actual cause.
2. Verify all servers run compatible Nomad versions (peersCache.ServersMeetMinimumVersion) before mixing key formats.
3. Confirm the fetched key bytes are intact — check for corruption between peers (TLS, disk).
4. If keyring state is corrupt, restore the data/keystore files from a healthy server backup and restart.
5. Restart the server so the keyring reinitializes and replication retries cleanly.

Example fix

// before: server on old version joins upgraded cluster
version = 1.5.x  // below minVersionKeyringInRaft
// after
upgrade server binary to match cluster >= minVersionKeyringInRaft, then restart

if !peersCache.ServersMeetMinimumVersion(region, minVersionKeyringInRaft, true) {
	return fmt.Errorf("cluster below min version for raft keyring; upgrade all servers first")
}

When it happens

Trigger: replicateKey gets a valid key from a peer but AddUnwrappedKey rejects it: bad key length/type for the AEAD cipher, the key already exists in an incompatible form, or the local keyring is in a bad state after a downgrade.

Common situations: Version skew: cluster upgraded past minVersionKeyringInRaft with mismatched key formats; corrupted in-memory keyring after a partial restore; key material from peers failing local cipher validation (same family as the 'could not configure cipher' error).

Related errors

• failed to fetch key from any peer: %v
• unable to decrypt wrapped key
• failed to configure keyring: %v
• failed to get active nomad key: %w
• only one server.keyring can be active in Nomad Community Edi

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/c1c760cd823b15c1. Report an issue: GitHub.