kubernetes/kops · error

error applying annotation to namespace: %v

Error message

error applying annotation to namespace: %v

What it means

After building the patch, SetInstalledVersion sends a StrategicMergePatch to the namespace to set the version annotation. If the Patch API call fails, the client error is wrapped as 'error applying annotation to namespace'. The version was not recorded even though the namespace exists.

Source

Thrown at channels/pkg/channels/channel_version.go:209

		return fmt.Errorf("error querying namespace %q: %v", c.Namespace, err)
	}

	value, err := version.Encode()
	if err != nil {
		return err
	}

	annotationPatch := &annotationPatch{Metadata: annotationPatchMetadata{Annotations: map[string]string{c.AnnotationName(): value}}}
	annotationPatchJSON, err := json.Marshal(annotationPatch)
	if err != nil {
		return fmt.Errorf("error building annotation patch: %v", err)
	}

	klog.V(2).Infof("sending patch: %q", string(annotationPatchJSON))

	_, err = k8sClient.CoreV1().Namespaces().Patch(ctx, c.Namespace, types.StrategicMergePatchType, annotationPatchJSON, metav1.PatchOptions{})
	if err != nil {
		return fmt.Errorf("error applying annotation to namespace: %v", err)
	}
	return nil
}

View on GitHub (pinned to 4c8573c808)

Solutions

1. Grant patch permission on namespaces to the caller's RBAC identity.
2. Re-check the namespace still exists; recreate if deleted.
3. Retry on transient/conflict errors (409/429/5xx).
4. Check admission webhooks or validating policies blocking namespace annotation updates.
5. Read the wrapped %v error to distinguish Forbidden vs NotFound vs server errors.

Example fix

// before
err := ch.SetInstalledVersion(ctx, client, version)
// after
err := retry.OnError(retry.DefaultRetry, apierrors.IsInternalError, func() error {
    return ch.SetInstalledVersion(ctx, client, version)
})

ssar, err := k8sClient.AuthorizationV1().SelfSubjectAccessReviews().Create(ctx, &authv1.SelfSubjectAccessReview{
    Spec: authv1.SelfSubjectAccessReviewSpec{ResourceAttributes: &authv1.ResourceAttributes{
        Verb: "patch", Resource: "namespaces", Name: namespace,
    }},
}, metav1.CreateOptions{})
// ssar.Status.Allowed must be true before calling SetInstalledVersion

func isRetryableAPIErr(err error) bool {
    return apierrors.IsConflict(err) || apierrors.IsTooManyRequests(err) || apierrors.IsInternalError(err)
}

err := retry.OnError(retry.DefaultBackoff, isRetryableAPIErr, func() error {
    return ch.SetInstalledVersion(ctx, k8sClient, version)
})
if err != nil && apierrors.IsForbidden(err) {
    return fmt.Errorf("RBAC: grant patch on namespaces: %w", err)
}

When it happens

Trigger: Calling Channel.SetInstalledVersion when the caller lacks patch permission on namespaces (Forbidden), the namespace was deleted between the GET and PATCH (NotFound), the patch payload is invalid (422), or the API server errors/conflicts.

Common situations: Service accounts with get-but-not-patch RBAC on namespaces; namespace concurrently deleted; API server admission webhooks rejecting the patch; transient API server issues.

Related errors

• error applying annotation to record addon installation: %v
• applying patch to node: %w
• error applying patch to node: %v
• error adding needs-update label: %v
• error patching needs-update label: %v

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/b567932bf5623b05. Report an issue: GitHub.