k3s-io/k3s · error

failed to update secret: %v

Error message

failed to update secret: %v

What it means

During the re-encrypt-active stage, k3s pages through every secret in every namespace and rewrites it so it is stored under the current encryption configuration. Each Update call that fails with a non-conflict error (conflicts are deliberately ignored because another writer won the race) aborts the whole re-encryption run with this wrapped error, after emitting a SecretsUpdateErrorEvent on the node.

Source

Thrown at pkg/server/handlers/secrets-encrypt.go:471

	}

	// For backwards compatibility with the old controller, we use an event recorder instead of logrus
	recorder := util.BuildControllerEventRecorder(ctx, k8s, "secrets-reencrypt", metav1.NamespaceDefault)

	secretPager := pager.New(pager.SimplePageFunc(func(opts metav1.ListOptions) (runtime.Object, error) {
		return k8s.CoreV1().Secrets(metav1.NamespaceAll).List(ctx, opts)
	}))
	secretPager.PageSize = secretsencrypt.SecretListPageSize

	i := 0
	if err := secretPager.EachListItem(ctx, metav1.ListOptions{}, func(obj runtime.Object) error {
		secret, ok := obj.(*corev1.Secret)
		if !ok {
			return errors.New("failed to convert object to Secret")
		}
		if _, err := k8s.CoreV1().Secrets(secret.Namespace).Update(ctx, secret, metav1.UpdateOptions{}); err != nil && !apierrors.IsConflict(err) {
			recorder.Eventf(nodeRef, corev1.EventTypeWarning, secretsencrypt.SecretsUpdateErrorEvent, "failed to update secret: %v", err)
			return fmt.Errorf("failed to update secret: %v", err)
		}
		if i != 0 && i%50 == 0 {
			recorder.Eventf(nodeRef, corev1.EventTypeNormal, secretsencrypt.SecretsProgressEvent, "reencrypted %d secrets", i)
		}
		i++
		return nil
	}); err != nil {
		return err
	}
	recorder.Eventf(nodeRef, corev1.EventTypeNormal, secretsencrypt.SecretsUpdateCompleteEvent, "reencrypted %d secrets", i)
	return nil
}

func AppendNewEncryptionKey(keys *secretsencrypt.EncryptionKeys, keyType string) error {
	var keyPrefix string
	switch keyType {
	case secretsencrypt.AESCBCProvider:
		keyPrefix = "aescbckey-"

View on GitHub (pinned to 6ba341e396)

Solutions

1. Check the node events (SecretsUpdateErrorEvent) and apiserver logs for the underlying error; the %v payload names the real cause.
2. Fix the root cause: restore apiserver/etcd health, correct RBAC, or resolve the webhook/validation rejection for the named secret.
3. Rerun the stage - re-encryption is idempotent; already-rewritten secrets simply get updated again.
4. If a single secret is permanently un-writable, export it, fix or delete it, then rerun the stage.

// Pre-flight: apiserver reachable and no failing webhooks on secrets
_, err := k8s.CoreV1().Secrets("default").List(ctx, metav1.ListOptions{Limit: 1})
if err != nil { log.Fatal("apiserver not ready for re-encryption: ", err) }

// Retry the stage; conflicts are already tolerated internally, transient failures are re-runnable
for attempt := 1; attempt <= 3; attempt++ {
    if err := runReencrypt(ctx); err == nil { break } else if attempt == 3 { return err }
    time.Sleep(time.Duration(attempt) * 30 * time.Second)
}

When it happens

Trigger: secretPager.EachListItem invoking Secrets.Update returns a non-conflict API error: RBAC/authorization failure on secrets, request validation failure (immutable field or malformed secret), apiserver unavailable mid-run, or the etcd backend erroring. Any single failed secret stops the stage.

Common situations: Running re-encrypt with degraded apiserver/etcd; admission webhooks rejecting the no-op update; extremely large secrets exceeding etcd limits; network interruptions between the node and the apiserver during the multi-minute page loop.

Related errors

• password hash not found in node secret
• invalid flag use; cannot use --disable-apiserver with --data
• toleration with empty key must have operator 'Exists'
• toleration with operator 'Exists' must have an empty value
• unsupported encryption keys found

AI-assisted analysis of k3s-io/k3s@6ba341e396 (2026-08-15). Data as JSON: /api/errors/04a3e27e4b9cbcf7. Report an issue: GitHub.