kubernetes/kops · error

error getting Instance: %v

Error message

error getting Instance: %v

What it means

Returned by GetCloudGroups when fetching details of an individual member instance of a MIG via Compute().Instances().Get() fails with a non-not-found error. Not-found is intentionally tolerated (logged as a warning since the instance may not be created yet); any other API error aborts listing the cloud groups.

Source

Thrown at upup/pkg/fi/cloudup/gce/instancegroups.go:209

				MaxSize:       int(mig.TargetSize),
				Raw:           mig,
			}
			groups[mig.Name] = g

			latestInstanceTemplate := mig.InstanceTemplate

			instances, err := ListManagedInstances(c, mig)
			if err != nil {
				return nil, err
			}

			for _, i := range instances {
				id := i.Instance
				name := LastComponent(id)
				instance, err := c.Compute().Instances().Get(project, zoneName, name)
				if err != nil {
					if !IsNotFound(err) {
						return nil, fmt.Errorf("error getting Instance: %v", err)
					}
					klog.Warningf("Instance %s not found, it may not have been created", name)
					continue
				}
				cm := &cloudinstances.CloudInstance{
					ID:                 instance.SelfLink,
					CloudInstanceGroup: g,
				}
				addCloudInstanceData(cm, instance)

				// Try first by provider ID
				providerID := "gce://" + project + "/" + zoneName + "/" + name
				node := nodesByProviderID[providerID]

				if node != nil {
					cm.Node = node
				} else {
					klog.V(8).Infof("unable to find node for instance: %s", id)

View on GitHub (pinned to 4c8573c808)

Solutions

1. Read the wrapped cause: 403 -> restore compute.instances.get IAM permission for the kops service account.
2. Retry on 429/5xx — this is often transient rate limiting when listing many instances across zones.
3. Verify the instance exists in the expected zone (gcloud compute instances describe <name> --zone <zone>).
4. If the instance is a zombie (MIG says member, GCE says otherwise), recreate the MIG members via rolling-update.

Example fix

// before
return nil, fmt.Errorf("error getting Instance: %v", err)
// after
gerr, ok := err.(*googleapi.Error)
if ok && gerr.Code == 429 {
    return nil, fmt.Errorf("rate limited getting Instance %s; retry: %v", name, err)
}
return nil, fmt.Errorf("error getting Instance: %v", err)

// verify each MIG member is fetchable before bulk operations
for _, inst := range migMembers {
    if _, err := computeService.Instances.Get(project, zone, inst.Name).Do(); err != nil {
        log.Warnf("instance %s in zone %s not fetchable: %v", inst.Name, zone, err)
    }
}

func isNotFoundOrAbsent(err error) bool {
    var gerr *googleapi.Error
    return errors.As(err, &gerr) && (gerr.Code == 404 || gerr.Code == 400)
}

instance, err := c.Compute().Instances().Get(project, zone, name)
if err != nil {
    if isNotFoundOrAbsent(err) {
        log.Warnf("Instance %s not found, skipping", name)
        continue
    }
    if isGoogleAPIErr(err, 429) || isGoogleAPIErr(err, 500) {
        // backoff and retry the Get
    }
    return nil, fmt.Errorf("error getting Instance: %v", err)
}

When it happens

Trigger: Calling GetCloudGroups where Instances().Get(project, zoneName, name) errors on a MIG member: permission denied on compute.instances.get, invalid instance name derived from the self-link, API rate limiting, or transient GCE outage.

Common situations: Large clusters hitting GCE API rate limits during listing, IAM role changes removing compute.viewer, instances in the middle of creation/deletion returning unexpected errors, or corrupted instance self-links after manual console edits.

Related errors

• error listing InstanceGroupManagers: %v
• error getting ForwardingRule %q: %w
• error getting TargetPool %q: %w
• error listing Routes: %w
• error listing Addresses: %v

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/f06e2f71edb4106f. Report an issue: GitHub.