kubernetes/kops · error

error deleting HTTP HealthCheck %s: %v

Error message

error deleting HTTP HealthCheck %s: %v

What it means

Wraps a failure from Compute HTTPHealthChecks().Delete in deleteHTTPHealthCheck, used when tearing down cluster load-balancer resources. A 404 is treated as success ('assuming deleted'), so this error means the delete call itself failed for a non-NotFound reason. The message includes the health check's SelfLink and the raw underlying error (%v, not %w).

Source

Thrown at pkg/resources/gce/gce.go:700

// deleteHTTPHealthCheck is the helper function to delete a Resource for a HTTP health check object
func deleteHTTPHealthCheck(cloud fi.Cloud, r *resources.Resource) error {
	c := cloud.(gce.GCECloud)
	t := r.Obj.(*compute.HttpHealthCheck)

	klog.V(2).Infof("Deleting GCE HTTP HealthCheck %s", t.SelfLink)
	u, err := gce.ParseGoogleCloudURL(t.SelfLink)
	if err != nil {
		return err
	}

	op, err := c.Compute().HTTPHealthChecks().Delete(u.Project, u.Name)
	if err != nil {
		if gce.IsNotFound(err) {
			klog.Infof("HTTP HealthCheck not found, assuming deleted: %q", t.SelfLink)
			return nil
		}
		return fmt.Errorf("error deleting HTTP HealthCheck %s: %v", t.SelfLink, err)
	}

	return c.WaitForOp(op)
}

// deleteFirewallRule is the helper function to delete a Resource for a Firewall object
func deleteFirewallRule(cloud fi.Cloud, r *resources.Resource) error {
	c := cloud.(gce.GCECloud)
	t := r.Obj.(*compute.Firewall)

	klog.V(2).Infof("Deleting GCE FirewallRule %s", t.SelfLink)
	u, err := gce.ParseGoogleCloudURL(t.SelfLink)
	if err != nil {
		return err
	}

	op, err := c.Compute().Firewalls().Delete(u.Project, u.Name)
	if err != nil {

View on GitHub (pinned to 4c8573c808)

Solutions

1. Ensure the target pool referencing the health check is deleted first, then re-run kops delete cluster.
2. Check IAM: the service account needs compute.healthChecks.delete.
3. Verify no concurrent kops/cloud operations are deleting the same resource; re-run after the other completes.
4. If the GCP console shows the health check already gone, re-run — NotFound is treated as success.
5. For a stuck 412, inspect which target pool still references it and remove the reference.

// ensure nothing still references the health check
pool, err := computeService.TargetPools.Get(project, region, poolName).Do()
if err == nil {
    for _, hc := range pool.HealthChecks {
        if strings.HasSuffix(hc, "/healthChecks/"+hcName) {
            return fmt.Errorf("target pool %s still references health check %s", poolName, hcName)
        }
    }
}

func isGCEAPIError(err error) (*googleapi.Error, bool) {
    var gerr *googleapi.Error
    return gerr, errors.As(err, &gerr)
}

for attempt := 0; attempt < 5; attempt++ {
    err := deleteHTTPHealthCheck(t)
    var gerr *googleapi.Error
    if err == nil { break }
    if errors.As(err, &gerr) && gerr.Code == 404 { break } // already deleted
    if errors.As(err, &gerr) && (gerr.Code == 412 || gerr.Code == 429 || gerr.Code >= 500) {
        time.Sleep(backoff(attempt)); continue
    }
    return err // permanent
}

When it happens

Trigger: c.Compute().HTTPHealthChecks().Delete(u.Project, u.Name) returns 403 permission denied, 412 conflict (health check still in use by a target pool), 429 rate limit, or 5xx. The 412 in-use case is the most common non-trivial cause.

Common situations: Target pool still references the health check because target pool deletion failed or was skipped earlier in teardown; kops service account lost compute.healthChecks.delete permission; concurrent deletion attempts from two kops runs.

Related errors

• error deleting FirewallRule %s: %v
• error deleting Route %s: %v
• error deleting Address %s: %v
• creating gce IPAM controller: %w
• error finding CloudInstanceGroups: %v

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/1e090d797cf58dd6. Report an issue: GitHub.