{"record":{"id":"04a3e27e4b9cbcf7","repo":"k3s-io/k3s","slug":"failed-to-update-secret-v","errorCode":null,"errorMessage":"failed to update secret: %v","messagePattern":"failed to update secret: (.+?)","errorType":"http","errorClass":null,"httpStatus":400,"severity":"error","filePath":"pkg/server/handlers/secrets-encrypt.go","lineNumber":471,"sourceCode":"\t}\n\n\t// For backwards compatibility with the old controller, we use an event recorder instead of logrus\n\trecorder := util.BuildControllerEventRecorder(ctx, k8s, \"secrets-reencrypt\", metav1.NamespaceDefault)\n\n\tsecretPager := pager.New(pager.SimplePageFunc(func(opts metav1.ListOptions) (runtime.Object, error) {\n\t\treturn k8s.CoreV1().Secrets(metav1.NamespaceAll).List(ctx, opts)\n\t}))\n\tsecretPager.PageSize = secretsencrypt.SecretListPageSize\n\n\ti := 0\n\tif err := secretPager.EachListItem(ctx, metav1.ListOptions{}, func(obj runtime.Object) error {\n\t\tsecret, ok := obj.(*corev1.Secret)\n\t\tif !ok {\n\t\t\treturn errors.New(\"failed to convert object to Secret\")\n\t\t}\n\t\tif _, err := k8s.CoreV1().Secrets(secret.Namespace).Update(ctx, secret, metav1.UpdateOptions{}); err != nil && !apierrors.IsConflict(err) {\n\t\t\trecorder.Eventf(nodeRef, corev1.EventTypeWarning, secretsencrypt.SecretsUpdateErrorEvent, \"failed to update secret: %v\", err)\n\t\t\treturn fmt.Errorf(\"failed to update secret: %v\", err)\n\t\t}\n\t\tif i != 0 && i%50 == 0 {\n\t\t\trecorder.Eventf(nodeRef, corev1.EventTypeNormal, secretsencrypt.SecretsProgressEvent, \"reencrypted %d secrets\", i)\n\t\t}\n\t\ti++\n\t\treturn nil\n\t}); err != nil {\n\t\treturn err\n\t}\n\trecorder.Eventf(nodeRef, corev1.EventTypeNormal, secretsencrypt.SecretsUpdateCompleteEvent, \"reencrypted %d secrets\", i)\n\treturn nil\n}\n\nfunc AppendNewEncryptionKey(keys *secretsencrypt.EncryptionKeys, keyType string) error {\n\tvar keyPrefix string\n\tswitch keyType {\n\tcase secretsencrypt.AESCBCProvider:\n\t\tkeyPrefix = \"aescbckey-\"","sourceCodeStart":453,"sourceCodeEnd":489,"githubUrl":"https://github.com/k3s-io/k3s/blob/6ba341e396edc16b8dcae978a7c5e3ac7ee5606e/pkg/server/handlers/secrets-encrypt.go#L453-L489","documentation":"During the re-encrypt-active stage, k3s pages through every secret in every namespace and rewrites it so it is stored under the current encryption configuration. Each Update call that fails with a non-conflict error (conflicts are deliberately ignored because another writer won the race) aborts the whole re-encryption run with this wrapped error, after emitting a SecretsUpdateErrorEvent on the node.","triggerScenarios":"secretPager.EachListItem invoking Secrets.Update returns a non-conflict API error: RBAC/authorization failure on secrets, request validation failure (immutable field or malformed secret), apiserver unavailable mid-run, or the etcd backend erroring. Any single failed secret stops the stage.","commonSituations":"Running re-encrypt with degraded apiserver/etcd; admission webhooks rejecting the no-op update; extremely large secrets exceeding etcd limits; network interruptions between the node and the apiserver during the multi-minute page loop.","solutions":["Check the node events (SecretsUpdateErrorEvent) and apiserver logs for the underlying error; the %v payload names the real cause.","Fix the root cause: restore apiserver/etcd health, correct RBAC, or resolve the webhook/validation rejection for the named secret.","Rerun the stage - re-encryption is idempotent; already-rewritten secrets simply get updated again.","If a single secret is permanently un-writable, export it, fix or delete it, then rerun the stage."],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// Pre-flight: apiserver reachable and no failing webhooks on secrets\n_, err := k8s.CoreV1().Secrets(\"default\").List(ctx, metav1.ListOptions{Limit: 1})\nif err != nil { log.Fatal(\"apiserver not ready for re-encryption: \", err) }","typeGuard":null,"tryCatchPattern":"// Retry the stage; conflicts are already tolerated internally, transient failures are re-runnable\nfor attempt := 1; attempt <= 3; attempt++ {\n    if err := runReencrypt(ctx); err == nil { break } else if attempt == 3 { return err }\n    time.Sleep(time.Duration(attempt) * 30 * time.Second)\n}","preventionTips":["Ensure apiserver/etcd health before re-encrypt-active","Re-encryption is idempotent - rerun the stage after fixing the root cause","Watch node events for SecretsUpdateErrorEvent during the run"],"tags":["kubernetes","secrets","re-encryption","apiserver"],"backgroundTag":null,"analyzedSha":"6ba341e396edc16b8dcae978a7c5e3ac7ee5606e","analyzedAt":"2026-08-15T16:27:54.286Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}