kubernetes/kops · error

error deleting ServiceAccount

Error message

error deleting ServiceAccount %s: %v

What it means

Thrown by deleteServiceAccount when IAM().ServiceAccounts().Delete fails with any error other than NotFound. One of the cluster's generated service accounts could not be removed during cluster teardown.

Solutions

1. Grant roles/iam.serviceAccountAdmin to the kops service account.
2. Check the wrapped error; if it mentions keys/bindings, remove them (`gcloud iam service-accounts keys delete`, `gcloud projects remove-iam-policy-binding`).
3. Retry on 429/5xx; NotFound is already handled as success so re-running is safe.
4. Clean up leftovers manually: `gcloud iam service-accounts delete EMAIL`.

Example fix

// before
return fmt.Errorf("error deleting ServiceAccount %s: %v", o.Name, err)
// after
return fmt.Errorf("error deleting ServiceAccount %s: %w", o.Name, err)

// test delete permission without deleting: check roles via testIamPermissions
ok, err := iamClient.Projects.ServiceAccounts.TestIamPermissions(saName, &iamapi.TestIamPermissionsRequest{Permissions: []string{"iam.serviceAccounts.delete"}}).Do()

func isIAMPermissionError(err error) bool {
	ge, ok := err.(*googleapi.Error)
	return ok && ge.Code == 403
}

_, err := c.IAM().ServiceAccounts().Delete(o.Name)
if err != nil {
	if gce.IsNotFound(err) {
		return nil
	}
	var ge *googleapi.Error
	if errors.As(err, &ge) && ge.Code == 403 {
		klog.Warningf("missing iam.serviceAccounts.delete for %s", o.Name)
	}
	return fmt.Errorf("error deleting ServiceAccount %s: %w", o.Name, err)
}

When it happens

Trigger: ServiceAccounts().Delete(o.Name) returns 403 (caller lacks iam.serviceAccounts.delete, e.g. missing roles/iam.serviceAccountAdmin), 404-adjacent preconditions, 400 because the SA has user-managed keys or is disabled/deleted state conflicts, or a transient IAM API error.

Common situations: kops credentials lack the Service Account Admin role while having compute rights; SA still has active bindings/keys that some org policies require clearing first; IAM API throttling during teardown.

Related errors

• cannot apply changes to ServiceAccount
• created ServiceAccount did not have expected email; got
• error creating ServiceAccount
• error listing ServiceAccount
• error listing ServiceAccounts

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/16338700281f85c3. Report an issue: GitHub.

Appendix: source

Thrown at pkg/resources/gce/gce.go:1059

				break
			}
		}
	}
	return resourceTrackers, nil
}

func deleteServiceAccount(cloud fi.Cloud, r *resources.Resource) error {
	c := cloud.(gce.GCECloud)
	o := r.Obj.(*iam.ServiceAccount)

	klog.V(2).Infof("deleting GCE ServiceAccount %s", o.Name)
	_, err := c.IAM().ServiceAccounts().Delete(o.Name)
	if err != nil {
		if gce.IsNotFound(err) {
			klog.Infof("ServiceAccount not found, assuming deleted: %q", o.Name)
			return nil
		}
		return fmt.Errorf("error deleting ServiceAccount %s: %v", o.Name, err)
	}
	return nil
}

// containsOnlyListedIGMs returns true if all the given backend service's backends
// are contained in the provided list of IGM resources.
func containsOnlyListedIGMs(svc *compute.BackendService, igms []*resources.Resource) bool {
	if len(svc.Backends) == 0 {
		return false
	}

	for _, be := range svc.Backends {
		listed := false
		for _, igm := range igms {
			// NOTE: this should be sufficient / strict enough since IGM names include the cluster
			// that they are part of, but revisit if naming conventions change.
			if strings.HasSuffix(be.Group, "/"+igm.Name) {
				listed = true

View on GitHub (pinned to 4c8573c808)