kubernetes/kops · error

error getting ForwardingRule %q: %w

Error message

error getting ForwardingRule %q: %w

What it means

This error wraps a failure from the Google Compute Engine API when looking up a ForwardingRule by name during cluster resource discovery in listFirewallRules. For each k8s-related firewall rule, kOps resolves the forwarding rule that targets it; if the Compute API Get call fails with anything other than a 404 (which is handled gracefully as 'not a k8s rule'), the underlying GCP error is wrapped and discovery aborts. The %w wrapping preserves the original GCE API error for errors.Is/As inspection.

Source

Thrown at pkg/resources/gce/gce.go:589

		// find the objects if this is a Kubernetes LoadBalancer
		if strings.HasPrefix(firewallRule.Name, "k8s-fw-") {
			// We build a list of resources if this is a k8s firewall rule,
			// but we only add them once all the checks are complete
			var k8sResources []*resources.Resource

			k8sResources = append(k8sResources, firewallRuleResource)

			// We lookup the forwarding rule by name, but we then validate that it points to one of our resources
			forwardingRuleName := strings.TrimPrefix(firewallRule.Name, "k8s-fw-")
			forwardingRule, err := c.Compute().ForwardingRules().Get(ctx, c.Project(), c.Region(), forwardingRuleName)
			if err != nil {
				if gce.IsNotFound(err) {
					// We looked it up by name, so an error isn't unlikely
					klog.Warningf("could not find forwarding rule %q, assuming firewallRule %q is not a k8s rule", forwardingRuleName, firewallRule.Name)
					continue nextFirewallRule
				}
				return nil, fmt.Errorf("error getting ForwardingRule %q: %w", forwardingRuleName, err)
			}

			forwardingRuleResource := &resources.Resource{
				Name:    forwardingRule.Name,
				ID:      forwardingRule.Name,
				Type:    typeForwardingRule,
				Deleter: deleteForwardingRule,
				Obj:     forwardingRule,
			}
			if forwardingRule.Target != "" {
				forwardingRuleResource.Blocks = append(forwardingRuleResource.Blocks, typeTargetPool+":"+gce.LastComponent(forwardingRule.Target))
			}
			k8sResources = append(k8sResources, forwardingRuleResource)

			// TODO: Can we get k8s to set labels on the ForwardingRule?

			// TODO: Check description?  It looks like e.g. description: '{"kubernetes.io/service-name":"kube-system/guestbook"}'

View on GitHub (pinned to 4c8573c808)

Solutions

1. Re-run the kops command; transient 5xx/rate-limit errors usually resolve on retry (with backoff if quota-related).
2. Verify the service account has compute.forwardingRules.get permission (roles/compute.viewer or compute.networkAdmin on the project).
3. Check that the cluster's region in the cluster spec matches where the forwarding rules actually exist.
4. Inspect the wrapped error with errors.As to the Google API error to read the exact GCP status code (403 vs 429 vs 500).
5. If the region is wrong, fix spec.networking/region or point kops at the correct cluster.

// before listing, verify access
forwardingRule, err := computeService.ForwardingRules.Get(project, region, name).Do()
if err != nil {
    if gce.IsNotFound(err) { return nil } // benign: not a k8s rule
    if gerr, ok := err.(*googleapi.Error); ok {
        if gerr.Code == 403 { return fmt.Errorf("missing compute.forwardingRules.get IAM: %w", err) }
    }
    return err
}

func isGCEAPIError(err error) (*googleapi.Error, bool) {
    var gerr *googleapi.Error
    if errors.As(err, &gerr) { return gerr, true }
    return nil, false
}

resourceMap, err := listFirewallRules(ctx, c)
if err != nil {
    var gerr *googleapi.Error
    if errors.As(err, &gerr) && (gerr.Code == 429 || gerr.Code >= 500) {
        // transient: retry with backoff
    }
    return fmt.Errorf("firewall rule discovery failed: %w", err)
}

When it happens

Trigger: c.Compute().ForwardingRules().Get(project, region, forwardingRuleName) returns a non-NotFound error: API quota/exceeded rate limits, permission denied on compute.forwardingRules.get, transient 5xx from the Compute API, invalid region configuration, or network failure reaching the GCE endpoint.

Common situations: Service account missing compute.viewer role after credential rotation; GCE API quota exhaustion in a busy project; regional mismatch where the forwarding rule lives in a different region than c.Region(); transient GCP outages during cluster deletion; private DNS/proxy issues blocking googleapis.com.

Related errors

• error getting TargetPool %q: %w
• error listing Routes: %w
• error listing Addresses: %v
• error listing subnetworks: %v
• error listing InstanceGroupManagers: %v

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/78d95421055d89d9. Report an issue: GitHub.