kubernetes/kops · error

failed to list backend services: %w

Error message

failed to list backend services: %w

What it means

Thrown by listBackendServices when RegionBackendServices().List fails with a non-NotFound error. This listing is used (from listHealthchecks) to discover health checks referenced by the cluster's backend services during teardown.

Source

Thrown at pkg/resources/gce/gce.go:1098

		}

		if !listed {
			return false
		}
	}
	return true
}

func (d *clusterDiscoveryGCE) listBackendServices() ([]*resources.Resource, error) {
	c := d.gceCloud

	svcs, err := c.Compute().RegionBackendServices().List(context.Background(), c.Project(), c.Region())
	if err != nil {
		if gce.IsNotFound(err) {
			klog.Infof("BackendService not found, assuming none exist in project: %q region: %q", c.Project(), c.Region())
			return nil, nil
		}
		return nil, fmt.Errorf("failed to list backend services: %w", err)
	}
	// TODO: cache, for efficiency, if needed.
	// Find all relevant backend services by finding all the cluster's IGMs, and then
	// listing all backend services in the project / region, then selecting
	// the backend services which contain only the relevant IGMs.
	igms, err := d.listInstanceGroupManagersAndInstances()
	if err != nil {
		return nil, err
	}
	var bs []*resources.Resource
	for _, svc := range svcs {
		if containsOnlyListedIGMs(svc, igms) {
			resourceTracker := &resources.Resource{
				Name: svc.Name,
				ID:   svc.Name,
				Type: typeBackendService,
				Deleter: func(cloud fi.Cloud, r *resources.Resource) error {
					op, err := c.Compute().RegionBackendServices().Delete(c.Project(), c.Region(), svc.Name)

View on GitHub (pinned to 4c8573c808)

Solutions

1. Read the wrapped error for status code and details.
2. Verify the region matches the cluster spec and the project has the Compute API enabled.
3. Grant compute.networkViewer (backendServices.list) to the credentials.
4. Retry on transient 429/5xx errors.

Example fix

// before
return nil, fmt.Errorf("failed to list backend services: %w", err)
// after
return nil, fmt.Errorf("failed to list backend services in %s/%s: %w", c.Project(), c.Region(), err)

// preflight: region and API availability
svc, err := computeClient.RegionBackendServices.List(project, region).Do()
if err != nil {
	return fmt.Errorf("preflight backend service list failed in %s/%s: %w", project, region, err)
}

func isTransientAPIError(err error) bool {
	ge, ok := err.(*googleapi.Error)
	return ok && (ge.Code == 429 || ge.Code >= 500)
}

svcs, err := c.Compute().RegionBackendServices().List(ctx, project, region)
if err != nil {
	if gce.IsNotFound(err) {
		return nil, nil
	}
	if isTransientAPIError(err) {
		return retryWithBackoff(func() ([]*resources.Resource, error) { return listBackendServices(d) })
	}
	return nil, fmt.Errorf("failed to list backend services: %w", err)
}

When it happens

Trigger: RegionBackendServices().List(project, region) returns 403 (no compute.backendServices.list permission), invalid region, Compute API disabled/quota issues, or transient 5xx. NotFound is deliberately treated as 'none exist', so only genuine errors reach this message.

Common situations: Deletion credentials lacking compute viewer roles; cluster spec's region doesn't exist in the project; Compute API temporarily unavailable mid-teardown; cross-project credential confusion.

Related errors

• error getting ForwardingRule %q: %w
• error listing Routes: %w
• error listing Addresses: %v
• error listing subnetworks: %v
• error listing routers: %v

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/d02867572184ec49. Report an issue: GitHub.