hashicorp/nomad · error

rotated key does not exist in keyring: %w

Error message

rotated key does not exist in keyring: %w

What it means

During a keyring rotation (variablesRekey), the server wraps data with the new root key and then looks the key up via the encrypter. If GetKey returns an error, the rotated key ID is not present in the local keyring, meaning the cluster's keyrings are out of sync or the key was removed before rotation completed.

Source

Thrown at nomad/core_sched.go:1268

			return err
		}

		// Perform the re-encryption of variables using the new active key. If
		// we reach a timeout, there is no need to return an error, as a new
		// eval will be emitted to continue the work. We do not mark the key
		// as inactive until all variables have been rekeyed. If any other error
		// occurs, we return it to the caller.
		if err = c.rotateVariables(varIter, eval); err != nil {
			if errors.Is(err, context.DeadlineExceeded) {
				c.logger.Info("timeout reached rekeying variables", "key_id", wrappedKeys.KeyID)
				return nil
			}
			return err
		}

		rootKey, err := c.srv.encrypter.GetKey(wrappedKeys.KeyID)
		if err != nil {
			return fmt.Errorf("rotated key does not exist in keyring: %w", err)
		}
		rootKey = rootKey.MakeInactive()

		req := &structs.KeyringUpdateRootKeyRequest{
			RootKey: rootKey,
			WriteRequest: structs.WriteRequest{
				Region:    c.srv.config.Region,
				AuthToken: eval.LeaderACL,
			},
		}
		if err := c.srv.RPC("Keyring.Update",
			req, &structs.KeyringUpdateRootKeyResponse{}); err != nil {
			c.logger.Error("rekey complete but failed to mark key as inactive", "error", err)
			return err
		}

		// Log a success, so cluster operators can see that the rekey has
		// completed successfully.

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Verify the key exists with nomad operator root keyring list and re-run rotation after replication completes
2. Check keyring replication/network connectivity between servers
3. Restore the missing key into the keyring before retrying rotation

// precheck: ensure the key exists before rotation
if _, err := encrypter.GetKey(keyID); err != nil {
    return fmt.Errorf("skip rotation, key %s not in keyring", keyID)
}

// wrap and inspect
if err := rotateKeys(); err != nil {
    if strings.Contains(err.Error(), "does not exist in keyring") {
        // resync/retry after replication
    }
}

When it happens

Trigger: Calling the keyring rotation path when wrappedKeys.KeyID refers to a root key absent from the local keyring store, e.g. the key was deleted or the keyring replication lagged behind the leader that generated it.

Common situations: Running rotation on a server that has not yet replicated the new key; an operator concurrently removed the key from the keyring; restoring a server from a snapshot missing newer keys.

Related errors

• unable to decrypt wrapped key
• failed to configure keyring: %v
• failed to get active nomad key: %w
• %w (root key): %w
• could not add cipher: %w

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/ffacec58beaa1575. Report an issue: GitHub.