hashicorp/nomad · critical

%w (root key): %w

Error message

%w (root key): %w

What it means

Nomad's Encrypter failed to decrypt a wrapped root key's data-encryption key (DEK) using the KMS wrapper during keyring restoration. The error is wrapped as ErrDecryptFailed with the underlying crypto error, meaning the ciphertext could not be opened with the supplied root key material. It is retried with backoff first, since transient issues (e.g. KMS unavailability) are possible.

Source

Thrown at nomad/encrypter.go:577

// channel.
//
// The error returned is only for testing and diagnostics.
func (e *Encrypter) decryptWrappedKeyTask(
	ctx context.Context, wrapper kms.Wrapper, meta *structs.RootKeyMeta,
	wrappedKey *structs.WrappedKey, respCh chan *cipherSet) error {

	var key []byte
	var rsaKey []byte

	minBackoff := time.Second
	maxBackoff := time.Second * 5

	err := helper.WithBackoffFunc(ctx, minBackoff, maxBackoff, func() error {
		wrappedDEK := wrappedKey.WrappedDataEncryptionKey
		var err error
		key, err = wrapper.Decrypt(e.srv.shutdownCtx, wrappedDEK)
		if err != nil {
			err := fmt.Errorf("%w (root key): %w", ErrDecryptFailed, err)
			e.log.Error(err.Error(), "key_id", meta.KeyID)
			return err
		}
		return nil
	})
	if err != nil {
		return err
	}

	err = helper.WithBackoffFunc(ctx, minBackoff, maxBackoff, func() error {
		var err error

		// Decrypt RSAKey for Workload Identity JWT signing if one exists. Prior to
		// 1.7 an ed25519 key derived from the root key was used instead of an RSA
		// key.
		if wrappedKey.WrappedRSAKey != nil && len(wrappedKey.WrappedRSAKey.Ciphertext) > 0 {
			rsaKey, err = wrapper.Decrypt(e.srv.shutdownCtx, wrappedKey.WrappedRSAKey)
			if err != nil {

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Check the underlying wrapped error for the real cause (auth failure vs bad ciphertext vs KMS unreachable) and fix that dependency first
2. Verify the external KMS credentials/config used by the kms wrapper are valid and the key still exists
3. Confirm the wrapped key ciphertext belongs to the same root key (KeyID in meta matches); restore a consistent keyring snapshot if state was copied across clusters
4. If keyring state is unrecoverable, reinitialize the keyring (keyring rotate/bootstrap) — note this affects decryptability of previously encrypted data

Example fix

// no caller code fix; fix the KMS dependency
// before: wrapper can't authenticate to Vault Transit
// after: export VAULT_TOKEN=<valid token> and ensure transit key <key_id> exists, then restart Nomad agent

// before relying on decrypt: verify the external KMS is reachable/authorized
err := wrapper.Encrypt(ctx, []byte("probe"))
if err != nil { /* fix KMS credentials/connectivity first */ }

if err != nil {
    if errors.Is(err, ErrDecryptFailed) {
        // inspect wrapped cause; retry with backoff or re-rotate keyring
    }
}

When it happens

Trigger: wrapper.Decrypt(shutdownCtx, wrappedDEK) returns an error inside the backoff retry while restoring a keyring key — typically because the wrapped DEK ciphertext was produced by a different/rotated root key, or the external KMS (Vault Transit, AWS KMS, etc.) rejects the decrypt operation.

Common situations: Key material changed or was rotated out from under stored wrapped keys; external KMS credentials expired or the KMS is unreachable; keyring state was copied between clusters or environments with different root keys; corrupted raft state after a restore.

Related errors

• unable to decrypt wrapped key
• failed to configure keyring: %v
• failed to get active nomad key: %w
• rotated key does not exist in keyring: %w
• keyring is not ready - waiting for keys %s

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/0e40b5ac098af5db. Report an issue: GitHub.