error fetching GCE managed instance group members for %q: %v

Error message

error fetching GCE managed instance group members for %q: %v

What it means

While enumerating members of a GCE managed instance group, the paginated ListInstances call failed, so kops cannot match the kubelet instance ID to a GCE instance. This wraps the error from iterating MIG members and is distinct from the case where the list succeeds but contains no match.

Source

Thrown at pkg/nodeidentity/gce/identify.go:259

}

// getManagedInstance queries GCE for the instance from the MIG
func (i *nodeIdentifier) getManagedInstance(ctx context.Context, mig *compute.InstanceGroupManager, instanceID uint64) (*compute.ManagedInstance, error) {
	var matches []*compute.ManagedInstance

	filter := "id=" + strconv.FormatUint(instanceID, 10)
	zone := lastComponent(mig.Zone)
	if err := i.computeService.InstanceGroupManagers.ListManagedInstances(i.project, zone, mig.Name).Filter(filter).Pages(ctx, func(page *compute.InstanceGroupManagersListManagedInstancesResponse) error {
		// Post-filter... filters aren't implemented (b/27605549)
		for _, instance := range page.ManagedInstances {
			if instance.Id != instanceID {
				continue
			}
			matches = append(matches, instance)
		}
		return nil
	}); err != nil {
		return nil, fmt.Errorf("error fetching GCE managed instance group members for %q: %v", mig.Name, err)
	}

	if len(matches) == 0 {
		return nil, fmt.Errorf("instance %v not managed by mig %s", instanceID, mig.Name)
	}
	if len(matches) > 1 {
		// Should be impossible - shows that filters / post-filters are not working
		return nil, fmt.Errorf("found multiple instances with id %v managed by mig %s", instanceID, mig.Name)
	}

	return matches[0], nil
}

// lastComponent returns the last component of a URL, i.e. anything after the last slash
// If there is no slash, returns the whole string
func lastComponent(s string) string {
	lastSlash := strings.LastIndex(s, "/")
	if lastSlash != -1 {

View on GitHub (pinned to 4c8573c808)

Solutions

Retry; this call is often transient (quota or 5xx)
Check IAM for compute.instanceGroupManagers.listInstances on the project
Reduce API call pressure / increase quota for the controller's service account
Verify MIG and zone are correct via gcloud describe

Defensive patterns

Strategy: retry

Validate before calling

// ensure IAM + quota before listing
// gcloud projects get-iam-policy && check compute.instanceGroupManagers.listInstances

Try / catch

inst, err := getManagedInstance(mig, id)
if err != nil {
  if strings.Contains(err.Error(), "fetching GCE managed instance group members") { /* transient: retry with backoff */ }
  return err
}

Prevention

Enable exponential backoff on GCE API calls
Raise API quota for controllers managing large MIGs
Alert on 403s to catch IAM regressions early

When it happens

Trigger: The iterator over i.computeService.InstanceGroupManagers.ListInstances(project, zone, migName, filter) returns a non-nil error mid-pagination: rate limiting (429), transient 5xx, permission failure on compute.instanceGroupManagers.listInstances, or network interruption.

Common situations: Large MIGs requiring multiple pages hitting API quota; IAM changes removing list permission during operation; brief GCE API outage while a node registers.

Related errors

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/716cdd8d7aedb91f. Report an issue: GitHub.