{"record":{"id":"635ba2c09cb73583","repo":"hashicorp/nomad","slug":"keyring-is-not-ready-waiting-for-keys-s","errorCode":null,"errorMessage":"keyring is not ready - waiting for keys %s","messagePattern":"keyring is not ready - waiting for keys (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"nomad/encrypter.go","lineNumber":257,"sourceCode":"\n\t\tfor _, id := range basePendingTasks {\n\t\t\tif _, ok := e.decryptTasks[id]; ok {\n\t\t\t\tcurrentPendingTasks = append(currentPendingTasks, id)\n\t\t\t}\n\t\t}\n\n\t\t// If we have decryption tasks which are still running that we care\n\t\t// about, log about this as well as return an error. If key decryption\n\t\t// progresses over time, an operator will be able to identify any\n\t\t// long-running tasks. If the timeout is reached, the final error is\n\t\t// sent to the caller which identifies the tasks that are taking too\n\t\t// long.\n\t\tif l := len(currentPendingTasks); l > 0 {\n\n\t\t\te.log.Debug(\"waiting for keyring to be ready\",\n\t\t\t\t\"num_tasks\", l, \"key_ids\", currentPendingTasks)\n\n\t\t\treturn fmt.Errorf(\"keyring is not ready - waiting for keys %s\",\n\t\t\t\tstrings.Join(currentPendingTasks, \", \"))\n\t\t}\n\t\treturn nil\n\t})\n\tif err != nil {\n\t\treturn err\n\t}\n\treturn nil\n}\n\n// Encrypt encrypts the clear data with the cipher for the active root key, and\n// returns the cipher text (including the nonce), and the key ID used to encrypt\n// it\nfunc (e *Encrypter) Encrypt(cleartext []byte) ([]byte, string, error) {\n\tcs, err := e.activeCipherSet()\n\tif err != nil {\n\t\treturn nil, \"\", err\n\t}","sourceCodeStart":239,"sourceCodeEnd":275,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/encrypter.go#L239-L275","documentation":"The keyring does not yet contain all keys referenced by keyring metadata (currentPendingTasks is non-empty), so encryption/signing work is temporarily refused. This is an expected transient condition while remote/KMS-wrapped keys are still being decrypted after startup or rotation.","triggerScenarios":"keyringIsReady (called before variable encryption or claim signing) finds pending decrypt tasks — keys listed in keystore metadata whose wrapped key material has not yet been resolved via configured KMS providers.","commonSituations":"Server just started and is still unwrapping keys via slow cloud KMS (AWS KMS, Vault transit), KMS provider credentials missing so unwrap retries indefinitely, or a key was added on the leader while this server is still catching up.","solutions":["Wait and retry — the operation is retried with backoff and should succeed once keys unwrap","Check server logs for decrypt task errors for the listed key IDs (KMS auth failures, network to KMS endpoint)","Verify the KEK provider configuration (credentials, region, Vault token) for the listed keys","If a key can never be unwrapped (lost KMS key), restore it from backup or rotate away from it"],"exampleFix":"// before: KMS creds absent, key never unwraps, calls keep failing\n// after: supply the provider credentials so unwrap completes\n//   export AWS_REGION=us-east-1\n//   export AWS_ACCESS_KEY_ID=... AWS_SECRET_ACCESS_KEY=...\n// then restart nomad agent and let pending key tasks drain","handlingStrategy":"retry","validationCode":"// poll keyring readiness before dependent work\nfunc waitForKeys(cli *api.Client, timeout time.Duration) error {\n  deadline := time.Now().Add(timeout)\n  for time.Now().Before(deadline) {\n    if _, _, err := cli.Keyring().List(nil); err == nil { return nil }\n    time.Sleep(2 * time.Second)\n  }\n  return fmt.Errorf(\"keyring not ready after %s\", timeout)\n}","typeGuard":null,"tryCatchPattern":"// the server itself retries; callers should tolerate transient failure\nerr := retry.Do(func() error {\n  _, err := vars.Decrypt(ctx, blob)\n  if strings.Contains(err.Error(), \"keyring is not ready\") {\n    return retry.Delay(2*time.Second) // transient, retry\n  }\n  return retry.Unrecoverable(err)\n});","preventionTips":["Ensure KMS provider credentials are available before agent startup","Allow time for key unwrapping after restarts before running dependent jobs","Alert on persistent 'keyring is not ready' in server logs (means a key can never unwrap)","Keep KMS endpoints reachable from every server in the region"],"tags":["keyring","kms","transient","retry"],"backgroundTag":"keyring-not-ready","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}