hashicorp/nomad · error

no such KMS provider %q configured

Error message

no such KMS provider %q configured

What it means

AddWrappedKey encountered a wrapped key entry whose provider ID has no matching entry in e.providerConfigs (the server's configured KEK providers). The key cannot be unwrapped and the error is accumulated into a multierror for the whole call.

Source

Thrown at nomad/encrypter.go:478

	// Use a channel to receive the cipherSet from the decrypter goroutines. It
	// allows us to fan-out decryption tasks for HA in Nomad Enterprise.
	cipherSetCh := make(chan *cipherSet)

	// We will use the key ID to track the decrypt tasks for this key. Doing
	// this here means we can do this once per function call.
	e.decryptTasksLock.Lock()
	e.decryptTasks[wrappedKeys.KeyID] = struct{}{}
	e.decryptTasksLock.Unlock()

	for _, wrappedKey := range wrappedKeys.WrappedKeys {
		providerID := wrappedKey.ProviderID
		if providerID == "" {
			providerID = string(structs.KEKProviderAEAD)
		}

		provider, ok := e.providerConfigs[providerID]
		if !ok {
			err := fmt.Errorf("no such KMS provider %q configured", providerID)
			mErr = multierror.Append(mErr, err)
			continue
		}

		wrapper, err := e.newKMSWrapper(provider, wrappedKeys.KeyID, wrappedKey.KeyEncryptionKey)
		if err != nil {
			// the errors that bubble up from this library can be a bit opaque, so
			// make sure we wrap them with as much context as possible
			err := fmt.Errorf("unable to create KMS wrapper for provider %q: %w", providerID, err)
			mErr = multierror.Append(mErr, err)
			continue
		}

		// fan-out decryption tasks for HA in Nomad Enterprise. we can use the
		// key whenever any one provider returns a successful decryption.
		go e.decryptWrappedKeyTask(completeCtx, wrapper, wrappedKeys.Meta(), wrappedKey, cipherSetCh)
		decryptTasks++
	}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Add the missing KEK provider block (matching the quoted provider ID in the error) to the server config and restart
2. Check the provider ID spelling in the config against the ID recorded in the keystore metadata
3. If the provider was intentionally removed, first rotate keys so no wrapped keys remain under that provider
4. Verify all servers share an equivalent provider configuration so replication can unwrap keys everywhere

Example fix

// before: server config missing provider used by wrapped keys
// after: add the provider block
kms = {
  provider     = "awskms"
  kms_key_id   = "alias/nomad-key"
  region       = "us-east-1"
}

// verify every provider ID used by keystore metadata exists in server config
for _, id := range providerIDsInKeystoreMetadata {
  if _, ok := configuredKEKProviders[id]; !ok {
    return fmt.Errorf("keystore references provider %q which is not in server config", id)
  }
}

if err := srv.AddWrappedKey(wrappedKeys); err != nil {
  if strings.Contains(err.Error(), "no such KMS provider") {
    return fmt.Errorf("server config is missing a KEK provider referenced by the keystore — add the kms block and restart: %w", err)
  }
  return err
}

When it happens

Trigger: AddWrappedKey (called by restoreImpl, applyRootKeyMetaUpsert, applyWrappedRootKeysUpsert) looks up e.providerConfigs[providerID]; the keyring metadata references a provider (e.g. "awskms", "transit", "gcpckms", "azurekeyvault", or custom ID) that the agent config does not define, defaulting to "aead" when the ID is empty.

Common situations: Keyring replicated from a cluster whose servers define a cloud KMS provider but this server's config omits it, typo in the kek provider name in the server config block, or the provider block was removed after keys were wrapped under it.

Related errors

• unable to create KMS wrapper for provider %q: %w
• dynamic workload users disabled
• nil config passed
• nil logger passed
• plugin config passed without binary name

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/b4f113728396fc60. Report an issue: GitHub.