old ServiceAccount signing key not in new ServiceAccount key

Error message

old ServiceAccount signing key not in new ServiceAccount key list

What it means

caCertReplace in pkg/server/handlers/cert.go, when a new ServiceAccount signing key is supplied during rotate-ca, requires the current (old) signing key to appear in the new key file. oldKeys[0] is compared against every key in newKeys (reflect.DeepEqual); if absent it errors, because existing service account tokens would become invalid with no grace period for validation.

Source

Thrown at pkg/server/handlers/cert.go:254

// is also present in the new key list, to ensure that old signatures can still be validated.
func validateServiceKey(oldKeyPath, newKeyPath string) error {
	oldKeys, err := keyutil.PublicKeysFromFile(oldKeyPath)
	if err != nil {
		return err
	}

	newKeys, err := keyutil.PublicKeysFromFile(newKeyPath)
	if err != nil {
		return err
	}

	for _, key := range newKeys {
		if reflect.DeepEqual(oldKeys[0], key) {
			return nil
		}
	}

	return errors.New("old ServiceAccount signing key not in new ServiceAccount key list")
}

View on GitHub (pinned to 6ba341e396)

Solutions

Append the old key to the new key file: 'cat /var/lib/rancher/k3s/server/tls/service.key new-sa.key > sa-bundle.key' and pass that as --sa-key.
Confirm both keys parse: 'openssl pkey -in sa-bundle.key -pubout' twice (or count 'BEGIN PRIVATE KEY' blocks) before submitting.
After rotation completes cluster-wide, the old key can be retired in a later step.

Example fix

# before: only new key -> error
openssl genrsa -out sa.key 2048
k3s certificate rotate-ca --sa-key=sa.key

# after: old + new in one file
 cat /var/lib/rancher/k3s/server/tls/service.key sa.key > sa-bundle.key
k3s certificate rotate-ca --sa-key=sa-bundle.key

Defensive patterns

Strategy: validation

Validate before calling

// Ensure the old SA signing key is present in the new key file
oldPub, _ := keyutil.PublicKeysFromFile("/var/lib/rancher/k3s/server/tls/service.key")
newPubs, _ := keyutil.PublicKeysFromFile(saBundlePath)
found := false
for _, k := range newPubs {
    if reflect.DeepEqual(oldPub[0], k) {
        found = true
    }
}
if !found {
    return errors.New("bundle must include the current service.key")
}

Try / catch

if err != nil && strings.Contains(err.Error(), "not in new ServiceAccount key list") {
    // cat old service.key + new key into one file and retry rotate-ca
}

Prevention

Always append the existing service.key to the new SA key bundle before rotate-ca.
Count 'BEGIN PRIVATE KEY' blocks - it must be >= 2 during rotation.
Retire the old key only after every node trusts the new one.

When it happens

Trigger: 'k3s certificate rotate-ca' with a --sa-key file that contains only the brand-new key and not the key currently at /var/lib/rancher/k3s/server/tls/service.key. The check compares parsed public keys, so re-encoding the old key in a different format still matches - only omission fails.

Common situations: Generating a fresh SA key pair and pointing rotate-ca at it alone; forgetting that rotation must be additive for tokens to stay valid.

Related errors

AI-assisted analysis of k3s-io/k3s@6ba341e396 (2026-08-15). Data as JSON: /api/errors/6095a79ce5c92834. Report an issue: GitHub.