kubernetes/kops · error

error fetching GCE instance: %w

Error message

error fetching GCE instance: %w

What it means

getInstance wraps the GCE compute.Instances.Get call; any error from the API (not-found, permission denied, quota, network) is wrapped with this message. It means the identifier could not fetch the instance resource for the given project/zone/name, and the underlying GCE API error is preserved via %w for inspection.

Source

Thrown at pkg/nodeidentity/gce/identify.go:217

			case kops.InstanceGroupRoleAPIServer:
				labels[nodelabels.RoleLabelAPIServer16] = ""
			default:
				klog.Warningf("unknown node role %q for server %q", role, instance.SelfLink)
			}
		}
	}
	if igName != "" {
		labels[kops.NodeLabelInstanceGroup] = igName
	}
	info.Labels = labels
	return info, nil
}

// getInstance queries GCE for the instance with the specified name, returning an error if not found
func (i *nodeIdentifier) getInstance(zone string, instanceName string) (*compute.Instance, error) {
	instance, err := i.computeService.Instances.Get(i.project, zone, instanceName).Do()
	if err != nil {
		return nil, fmt.Errorf("error fetching GCE instance: %w", err)
	}

	return instance, nil
}

// getInstanceTemplate queries GCE for the IG Template with the specified name, returning an error if not found
func (i *nodeIdentifier) getInstanceTemplate(name string) (*compute.InstanceTemplate, error) {
	t, err := i.computeService.InstanceTemplates.Get(i.project, name).Do()
	if err != nil {
		return nil, fmt.Errorf("error fetching GCE instance group template %q: %v", name, err)
	}

	return t, nil
}

// getMIG queries GCE for the MIG with the specified name, returning an error if not found
func (i *nodeIdentifier) getMIG(zone string, migName string) (*compute.InstanceGroupManager, error) {
	mig, err := i.computeService.InstanceGroupManagers.Get(i.project, zone, migName).Do()

View on GitHub (pinned to 4c8573c808)

Solutions

1. Unwrap the error in logs: if 404, the instance is gone — delete the stale k8s Node object.
2. If 403/PERMISSION_DENIED, grant the controller's service account compute.instances.get (roles/compute.viewer) in the node project.
3. Ensure the Compute Engine API is enabled (gcloud services enable compute.googleapis.com).
4. For transient errors (5xx, timeouts, rate limits), add retry with exponential backoff around IdentifyNode/getInstance.

Example fix

// before
instance, err := i.computeService.Instances.Get(i.project, zone, instanceName).Do()
if err != nil { return nil, fmt.Errorf("error fetching GCE instance: %w", err) }
// after — retry transient failures
var instance *compute.Instance
err := wait.ExponentialBackoff(defaultBackoff, func() (bool, error) {
    inst, err := i.computeService.Instances.Get(i.project, zone, instanceName).Do()
    if err == nil { instance = inst; return true, nil }
    if isNotFound(err) { return false, err } // permanent
    return false, nil                         // transient: retry
})
if err != nil { return nil, fmt.Errorf("error fetching GCE instance: %w", err) }

// pre-flight: verify IAM + API before identification loop
_, err := computeSvc.Projects.Get(project).Do()
if err != nil {
    return fmt.Errorf("GCE API/IAM preflight failed for project %s: %w", project, err)
}

func isNotFound(err error) bool {
    var apiErr *googleapi.Error
    return errors.As(err, &apiErr) && apiErr.Code == 404
}

info, err := identifier.IdentifyNode(ctx, node)
if err != nil {
    var apiErr *googleapi.Error
    if errors.As(err, &apiErr) && errors.Is(err, errWrapped) && apiErr.Code == 404 {
        // instance gone: delete stale Node
        return deleteNode(node)
    }
    if apiErr != nil && (apiErr.Code >= 500 || apiErr.Code == 429) {
        return requeueWithBackoff(err) // transient: retry
    }
    return err
}

When it happens

Trigger: IdentifyNode calls i.getInstance(zone, instanceName) and the GCE API call fails: instance deleted between parse and fetch, wrong zone in providerID, service account lacking compute.instances.get, API disabled, or transient 5xx/network error.

Common situations: Node's VM was deleted (stale Node object) — googleapi 404; IAM service account missing roles/compute.viewer; Compute Engine API disabled in the project; firewall/proxy blocking googleapis.com; transient GCE API outages.

Related errors

• error fetching GCE instance group template %q: %v
• error fetching GCE managed instance group %q: %v
• error listing ServiceAccount %q: %w
• error creating ServiceAccount %q: %w
• providerID was not set for node %s

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/519e03d06f25550b. Report an issue: GitHub.