kubernetes/kops · error

error updating Healthcheck: %v

Error message

error updating Healthcheck: %v

What it means

In the GCE httphealthcheck task, after the HTTPHealthChecks().Update API call succeeds, kOps waits for the returned GCE operation to complete via WaitForOp. This error wraps any failure from that operation wait — i.e. the update request was accepted but the operation itself failed or errored on the GCE side.

Source

Thrown at upup/pkg/fi/cloudup/gcetasks/httphealthcheck.go:114

		if err := t.Cloud.WaitForOp(r); err != nil {
			return fmt.Errorf("error creating Healthcheck: %v", err)
		}
		h.SelfLink = r.TargetLink
	} else if changes.Port != nil || changes.RequestPath != nil {
		// Insert only applies these on create, so reconcile changes to an existing check with a separate Update.
		o := &compute.HttpHealthCheck{
			Name:        fi.ValueOf(e.Name),
			Port:        fi.ValueOf(e.Port),
			RequestPath: fi.ValueOf(e.RequestPath),
		}

		klog.V(4).Infof("Updating Healthcheck %q", o.Name)
		r, err := t.Cloud.Compute().HTTPHealthChecks().Update(t.Cloud.Project(), o.Name, o)
		if err != nil {
			return fmt.Errorf("error updating Healthcheck %q: %v", o.Name, err)
		}
		if err := t.Cloud.WaitForOp(r); err != nil {
			return fmt.Errorf("error updating Healthcheck: %v", err)
		}
	}
	return nil
}

type terraformHTTPHealthcheck struct {
	Name        string  `cty:"name"`
	Port        *int64  `cty:"port"`
	RequestPath *string `cty:"request_path"`
}

func (_ *HTTPHealthcheck) RenderTerraform(t *terraform.TerraformTarget, a, e, changes *HTTPHealthcheck) error {
	tf := &terraformHTTPHealthcheck{
		Name:        *e.Name,
		Port:        e.Port,
		RequestPath: e.RequestPath,
	}
	return t.RenderResource("google_compute_http_health_check", *e.Name, tf)

View on GitHub (pinned to 4c8573c808)

Solutions

1. Inspect the wrapped %v error text for the GCE operation error detail and fix the offending healthcheck field in the cluster spec
2. Retry `kops update cluster` — operation-poll failures are often transient
3. Check GCP Console > Compute Engine > Operations (or `gcloud compute operations list`) for the failed operation's message
4. Verify healthcheck values (port, requestPath, checkIntervalSec, timeoutSec) satisfy GCE constraints

Example fix

// before
hc, err := t.Cloud.Compute().HTTPHealthChecks().Get(t.Cloud.Project(), o.Name)
// adjust fields blindly
// after
hc, err := t.Cloud.Compute().HTTPHealthChecks().Get(t.Cloud.Project(), o.Name)
if err != nil { return fmt.Errorf("error getting Healthcheck %q: %v", o.Name, err) }
// reconcile fields against hc before issuing Update so the operation cannot fail

// Validate healthcheck fields before running update
func validateHealthcheck(hc *compute.HttpHealthCheck) error {
	if hc.Port < 1 || hc.Port > 65535 { return fmt.Errorf("invalid port %d", hc.Port) }
	if hc.CheckIntervalSec < 1 || hc.TimeoutSec >= hc.CheckIntervalSec { return fmt.Errorf("timeoutSec must be < checkIntervalSec") }
	if hc.UnhealthyThreshold < 1 { return fmt.Errorf("unhealthyThreshold must be >= 1") }
	return nil
}

if err := t.Cloud.WaitForOp(r); err != nil {
	if isTransientOpError(err) {
		// re-fetch task state and retry the update with backoff
		return retryAfterBackoff(func() error { return task.Update(ctx) })
	}
	return fmt.Errorf("error updating Healthcheck: %v", err)
}

When it happens

Trigger: HTTPHealthChecks().Update succeeded but Cloud.WaitForOp(r) returned an error: the GCE operation completed with an error state (e.g. invalid field rejected asynchronously, quota issue, or a transient API failure while polling).

Common situations: GCE rejecting healthcheck fields asynchronously (port/path/checkInterval changes conflicting with backend services), transient Google API errors or rate limiting during operation polling, or project-level quotas preventing the update.

Related errors

• error building node identifier: %w
• creating gce IPAM controller: %w
• --project cannot be empty; specify a project or omit the fla
• error reading certificate %q: %v
• error listening on %q: %v

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/7044e4d868c58e3d. Report an issue: GitHub.