What causes "no such provider %q configured" in hashicorp/nomad?

Add the missing kms_config block (same provider and ID) back to the server configuration and restart.

no such provider %q configured

Error message

no such provider %q configured

What it means

loadKeyFromStore restores a key from the keyring store: it reads the stored KEKWrapper and looks up its provider by ProviderID in e.providerConfigs. If that provider ID is not configured on this server, it returns "no such provider %q configured". The stored key cannot be decrypted because the KEK needed to unwrap it is not available.

Source

Thrown at nomad/encrypter.go:951

		return nil, err
	}

	kekWrapper := &structs.KeyEncryptionKeyWrapper{}
	if err := json.Unmarshal(raw, kekWrapper); err != nil {
		return nil, err
	}

	meta := kekWrapper.Meta
	if err = meta.Validate(); err != nil {
		return nil, err
	}

	if kekWrapper.ProviderID == "" {
		kekWrapper.ProviderID = string(structs.KEKProviderAEAD)
	}
	provider, ok := e.providerConfigs[kekWrapper.ProviderID]
	if !ok {
		return nil, fmt.Errorf("no such provider %q configured", kekWrapper.ProviderID)
	}

	// the errors that bubble up from this library can be a bit opaque, so make
	// sure we wrap them with as much context as possible
	wrapper, err := e.newKMSWrapper(provider, meta.KeyID, kekWrapper.KeyEncryptionKey)
	if err != nil {
		return nil, fmt.Errorf("unable to create key wrapper: %w", err)
	}
	wrappedDEK := kekWrapper.WrappedDataEncryptionKey
	if wrappedDEK == nil {
		// older KEK wrapper versions with AEAD-only have the key material in a
		// different field
		wrappedDEK = &kms.BlobInfo{Ciphertext: kekWrapper.EncryptedDataEncryptionKey}
	}
	key, err := wrapper.Decrypt(e.srv.shutdownCtx, wrappedDEK)
	if err != nil {
		return nil, fmt.Errorf("%w (root key): %w", ErrDecryptFailed, err)
	}

View on GitHub (pinned to 482b49bf1a)

Solutions

Add the missing kms_config block (same provider and ID) back to the server configuration and restart.
List the stored keyring metadata to see which provider ID the keys expect, and configure exactly that.
If the old provider is intentionally retired, rotate to a new KEK/provider first, then remove the old config.
For keys restored from a different cluster, ensure both clusters share the same KMS provider configuration.

Example fix

// before: server config lacks the provider used by stored keys
// after
server {
  default_scheduler_config { }
}
kms_config {
  provider = "awskms"
  config { key_id = "alias/nomad-default" }
}

Defensive patterns

Strategy: validation

Validate before calling

// before restart/removal of a kms_config, verify no stored keys reference it:
// list keyring key metadata and assert each ProviderID exists in providerConfigs
for _, meta := range storedKeyMetas {
    if _, ok := providerConfigs[meta.ProviderID]; !ok {
        return fmt.Errorf("config would orphan key %s (provider %s)", meta.KeyID, meta.ProviderID)
    }
}

Type guard

func providerConfigured(id string, configs map[string]struct{}) bool {
    if id == "" { id = "aead" }
    _, ok := configs[id]
    return ok
}

Try / catch

key, err := loadKeyFromStore(...)
if err != nil && strings.Contains(err.Error(), "no such provider") {
    // restore the missing kms_config block or rotate keys before proceeding
}

Prevention

Never remove a kms_config block until all keys it wraps are rotated to a new provider.
Keep a record (config-as-code) of which provider IDs originally wrapped the keyring.
Diff provider config before and after upgrades or cluster migrations.

When it happens

Trigger: Restoring/decrypting a stored KEKWrapper whose ProviderID (defaulting to "aead" when empty) has no matching entry in the server's configured kms_config blocks.

Common situations: Server restarted without the kms_config block that originally wrapped the keyring; keys migrated between clusters with different KMS providers; provider renamed/removed in config; upgrading after changing providers without rotating keys.

Related errors

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/74f9ffdfa2ea4247. Report an issue: GitHub.