hashicorp/nomad · critical · ErrDecryptFailed

unable to decrypt wrapped key

Error message

unable to decrypt wrapped key

What it means

ErrDecryptFailed is the sentinel error returned when Nomad's Encrypter cannot decrypt a wrapped data-encryption key (or wrapped RSA key) using the keyring wrapper. At nomad/encrypter.go:577 and :596 the sentinel wraps the underlying Decrypt error, producing messages like 'unable to decrypt wrapped key (root key): ...'. It usually means the server cannot access the root key material (KMS) needed to unwrap per-key DEKs.

Source

Thrown at nomad/encrypter.go:997

		}
	} else if len(kekWrapper.EncryptedRSAKey) > 0 {
		// older KEK wrapper versions with AEAD-only have the key material in a
		// different field
		rsaKey, err = wrapper.Decrypt(e.srv.shutdownCtx, &kms.BlobInfo{
			Ciphertext: kekWrapper.EncryptedRSAKey})
		if err != nil {
			return nil, fmt.Errorf("%w (rsa key): %w", ErrDecryptFailed, err)
		}
	}

	return &structs.UnwrappedRootKey{
		Meta:   meta,
		Key:    key,
		RSAKey: rsaKey,
	}, nil
}

var ErrDecryptFailed = errors.New("unable to decrypt wrapped key")

// waitForPublicKey returns the public signing key for the requested key id or
// an error if the key could not be found. It blocks up to 1s for key material
// to be decrypted so that Workload Identities signed by a brand-new key can be
// verified for stale RPCs made to followers that might not have yet decrypted
// the key received via Raft
func (e *Encrypter) waitForPublicKey(keyID string) (*structs.KeyringPublicKey, error) {
	ctx, cancel := context.WithTimeout(e.srv.shutdownCtx, 1*time.Second)
	defer cancel()
	ks, err := e.waitForKey(ctx, keyID)
	if err != nil {
		return nil, err
	}

	pubKey := &structs.KeyringPublicKey{
		KeyID:      keyID,
		Use:        structs.PubKeyUseSig,
		CreateTime: ks.rootKey.Meta.CreateTime,

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Read the wrapped '%w (root key)' or '(rsa key)' suffix in the log for the underlying KMS error and fix that cause first
2. Verify the server's KMS credentials/permissions can decrypt the root key referenced by key_id
3. Restore or re-add the missing key with 'nomad keyring' operations (rotate/publish) once KMS access is fixed
4. Check that e.srv.shutdownCtx was not already cancelled (server shutting down) — retry after startup

Example fix

// before: server starts without KMS creds
export VAULT_TOKEN=  # expired
// after: provide valid KMS credentials before starting nomad
export VAULT_TOKEN=<valid-token>
systemctl restart nomad

// shell: verify KMS reachability before starting nomad
vault token lookup >/dev/null && echo 'kms ok' || { echo 'fix KMS credentials first'; exit 1; }

// Go
func isDecryptFailed(err error) bool {
    return errors.Is(err, nomad.ErrDecryptFailed) // sentinel-wrapped: %w chaining
}

// Go
key, err := e.loadKeyFromStore(meta)
if err != nil {
    if errors.Is(err, ErrDecryptFailed) {
        logger.Error("keyring decryption failed; check KMS creds for key", "key_id", meta.KeyID)
        alertKMSOperator(meta.KeyID)
        return err // do not proceed with undecryptable keys
    }
    return err
}

When it happens

Trigger: loadKeyFromStore unwrapping a stored wrappedDEK at startup or on key access; wrapper.Decrypt failing due to missing/unavailable KMS credentials, a key removed or rotated in the external KMS, or a corrupted keyring entry in the Raft-backed keyring store.

Common situations: KMS credentials (cloud IAM, Vault transit token) revoked or expired on the Nomad server; the root key in the external KMS was deleted; restoring Raft data onto a server without the same KMS access; entropy or shutdown context cancellation mid-decrypt.

Related errors

• failed to configure keyring: %v
• failed to get active nomad key: %w
• %w (root key): %w
• failed to add key to keyring: %v
• variable error: encrypt: %w

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/db2bcbdc54a1c5d7. Report an issue: GitHub.