kubernetes/kops · error

error recreating Instance %s: %v

Error message

error recreating Instance %s: %v

What it means

This error is returned by recreateCloudInstance when the GCE Compute API call InstanceGroupManagers().RecreateInstances() fails for a MIG member instance. kOps issues a recreateInstances request on the managed instance group to recreate a specific instance (e.g. during a rolling replace); any API error other than not-found is wrapped with the instance ID and the underlying cause. A not-found error is deliberately swallowed and treated as already deleted.

Source

Thrown at upup/pkg/fi/cloudup/gce/instancegroups.go:113

// recreateCloudInstance recreates the specified instances, managed by an InstanceGroupManager
func recreateCloudInstance(c GCECloud, i *cloudinstances.CloudInstance) error {
	mig := i.CloudInstanceGroup.Raw.(*compute.InstanceGroupManager)

	klog.V(2).Infof("Recreating GCE Instance %s in MIG %s", i.ID, mig.Name)

	migURL, err := ParseGoogleCloudURL(mig.SelfLink)
	if err != nil {
		return err
	}

	op, err := c.Compute().InstanceGroupManagers().RecreateInstances(migURL.Project, migURL.Zone, migURL.Name, i.ID)
	if err != nil {
		if IsNotFound(err) {
			klog.Infof("Instance not found, assuming deleted: %q", i.ID)
			return nil
		}
		return fmt.Errorf("error recreating Instance %s: %v", i.ID, err)
	}

	return c.WaitForOp(op)
}

// GetCloudGroups returns a map of CloudGroup that backs a list of instance groups
func (c *gceCloudImplementation) GetCloudGroups(cluster *kops.Cluster, instancegroups []*kops.InstanceGroup, warnUnmatched bool, nodes []v1.Node) (map[string]*cloudinstances.CloudInstanceGroup, error) {
	return GetCloudGroups(c, cluster, instancegroups, warnUnmatched, nodes)
}

func GetCloudGroups(c GCECloud, cluster *kops.Cluster, instancegroups []*kops.InstanceGroup, warnUnmatched bool, nodes []v1.Node) (map[string]*cloudinstances.CloudInstanceGroup, error) {
	groups := make(map[string]*cloudinstances.CloudInstanceGroup)

	project := c.Project()
	ctx := context.Background()

	nodesByProviderID := make(map[string]*v1.Node)

View on GitHub (pinned to 4c8573c808)

Solutions

1. Read the wrapped %v cause: if it's a GCE 4xx about the MIG or instance, verify the instance still exists and belongs to the MIG (gcloud compute instance-groups managed describe).
2. Check IAM: ensure the kOps service account has roles/compute.instanceAdmin.v1 (recreateInstances permission).
3. If the instance was already deleted externally, re-run; kOps may now see it gone and skip recreation.
4. Retry on transient Google API errors (5xx / rate limits); rolling-update commands are typically safe to re-run.
5. Confirm project/zone/name in the MIG URL match the cluster's cloud config.

Example fix

// before
return fmt.Errorf("error recreating Instance %s: %v", i.ID, err)
// after (diagnose the cause before retrying)
if gerr, ok := err.(*googleapi.Error); ok && gerr.Code == 404 {
    klog.Infof("MIG or instance no longer exists: %q", i.ID)
    return nil
}
return fmt.Errorf("error recreating Instance %s: %v", i.ID, err)

// before calling rolling-update/delete
migExists, _ := gcloudInstanceGroupExists(project, zone, migName)
instanceExists, _ := gcloudInstanceExists(project, zone, instanceName)
if !migExists || !instanceExists {
    // skip recreation; instance/MIG already gone
}

func isNotFoundErr(err error) bool {
    if gerr, ok := err.(*googleapi.Error); ok {
        return gerr.Code == 404
    }
    return false
}

result, err := recreateInstance(project, zone, migName, instanceID)
var gerr *googleapi.Error
if errors.As(err, &gerr) {
    switch {
    case gerr.Code == 404:
        log.Info("instance/MIG gone; treating as deleted")
    case gerr.Code == 429 || gerr.Code >= 500:
        retryWithBackoff(recreateInstance)
    default:
        log.Errorf("recreate failed for %s: %v", instanceID, err)
    }
}

When it happens

Trigger: Calling DeleteInstance -> recreateCloudInstance for an instance whose ID yields a RecreateInstances API error: MIG not found at migURL (project/zone/name), instance not a member of the MIG, GCE quota/regional outage, invalid or stale instance ID, or permission denied on the compute.instanceGroupManagers.updateInstances permission.

Common situations: Rolling-update/replace of a GCE cluster where the instance was concurrently deleted, the MIG was removed out-of-band, the service account lacks compute admin IAM roles, or a transient GCE API failure occurs during `kops rolling-update cluster`.

Related errors

• error listing InstanceGroupManagers: %v
• error getting instance group for MIG %q
• found multiple instance groups matching MIG %q
• instance group %q must specify exactly one zone
• instance group %q neither control-plane nor api-server

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/05ed76f49d288f48. Report an issue: GitHub.