{"record":{"id":"db2bcbdc54a1c5d7","repo":"hashicorp/nomad","slug":"unable-to-decrypt-wrapped-key","errorCode":null,"errorMessage":"unable to decrypt wrapped key","messagePattern":"unable to decrypt wrapped key","errorType":"exception","errorClass":"ErrDecryptFailed","httpStatus":null,"severity":"critical","filePath":"nomad/encrypter.go","lineNumber":997,"sourceCode":"\t\t}\n\t} else if len(kekWrapper.EncryptedRSAKey) > 0 {\n\t\t// older KEK wrapper versions with AEAD-only have the key material in a\n\t\t// different field\n\t\trsaKey, err = wrapper.Decrypt(e.srv.shutdownCtx, &kms.BlobInfo{\n\t\t\tCiphertext: kekWrapper.EncryptedRSAKey})\n\t\tif err != nil {\n\t\t\treturn nil, fmt.Errorf(\"%w (rsa key): %w\", ErrDecryptFailed, err)\n\t\t}\n\t}\n\n\treturn &structs.UnwrappedRootKey{\n\t\tMeta:   meta,\n\t\tKey:    key,\n\t\tRSAKey: rsaKey,\n\t}, nil\n}\n\nvar ErrDecryptFailed = errors.New(\"unable to decrypt wrapped key\")\n\n// waitForPublicKey returns the public signing key for the requested key id or\n// an error if the key could not be found. It blocks up to 1s for key material\n// to be decrypted so that Workload Identities signed by a brand-new key can be\n// verified for stale RPCs made to followers that might not have yet decrypted\n// the key received via Raft\nfunc (e *Encrypter) waitForPublicKey(keyID string) (*structs.KeyringPublicKey, error) {\n\tctx, cancel := context.WithTimeout(e.srv.shutdownCtx, 1*time.Second)\n\tdefer cancel()\n\tks, err := e.waitForKey(ctx, keyID)\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\n\tpubKey := &structs.KeyringPublicKey{\n\t\tKeyID:      keyID,\n\t\tUse:        structs.PubKeyUseSig,\n\t\tCreateTime: ks.rootKey.Meta.CreateTime,","sourceCodeStart":979,"sourceCodeEnd":1015,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/encrypter.go#L979-L1015","documentation":"ErrDecryptFailed is the sentinel error returned when Nomad's Encrypter cannot decrypt a wrapped data-encryption key (or wrapped RSA key) using the keyring wrapper. At nomad/encrypter.go:577 and :596 the sentinel wraps the underlying Decrypt error, producing messages like 'unable to decrypt wrapped key (root key): ...'. It usually means the server cannot access the root key material (KMS) needed to unwrap per-key DEKs.","triggerScenarios":"loadKeyFromStore unwrapping a stored wrappedDEK at startup or on key access; wrapper.Decrypt failing due to missing/unavailable KMS credentials, a key removed or rotated in the external KMS, or a corrupted keyring entry in the Raft-backed keyring store.","commonSituations":"KMS credentials (cloud IAM, Vault transit token) revoked or expired on the Nomad server; the root key in the external KMS was deleted; restoring Raft data onto a server without the same KMS access; entropy or shutdown context cancellation mid-decrypt.","solutions":["Read the wrapped '%w (root key)' or '(rsa key)' suffix in the log for the underlying KMS error and fix that cause first","Verify the server's KMS credentials/permissions can decrypt the root key referenced by key_id","Restore or re-add the missing key with 'nomad keyring' operations (rotate/publish) once KMS access is fixed","Check that e.srv.shutdownCtx was not already cancelled (server shutting down) — retry after startup"],"exampleFix":"// before: server starts without KMS creds\nexport VAULT_TOKEN=  # expired\n// after: provide valid KMS credentials before starting nomad\nexport VAULT_TOKEN=<valid-token>\nsystemctl restart nomad","handlingStrategy":"try-catch","validationCode":"// shell: verify KMS reachability before starting nomad\nvault token lookup >/dev/null && echo 'kms ok' || { echo 'fix KMS credentials first'; exit 1; }","typeGuard":"// Go\nfunc isDecryptFailed(err error) bool {\n    return errors.Is(err, nomad.ErrDecryptFailed) // sentinel-wrapped: %w chaining\n}","tryCatchPattern":"// Go\nkey, err := e.loadKeyFromStore(meta)\nif err != nil {\n    if errors.Is(err, ErrDecryptFailed) {\n        logger.Error(\"keyring decryption failed; check KMS creds for key\", \"key_id\", meta.KeyID)\n        alertKMSOperator(meta.KeyID)\n        return err // do not proceed with undecryptable keys\n    }\n    return err\n}","preventionTips":["Monitor KMS credential expiry and IAM policy changes","Never delete/rotate root keys in the external KMS while wrapped DEKs reference them","Test restores of Raft data against servers with matching KMS access","Alert on ErrDecryptFailed in server logs (key_id is logged alongside)"],"tags":["nomad","encryption","kms","keyring"],"backgroundTag":"key-decryption-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}