kubernetes/kops · error

error updating InstanceTemplate for InstanceGroupManager: %v

Error message

error updating InstanceTemplate for InstanceGroupManager: %v

What it means

kOps wraps a failure from the GCE Compute API SetInstanceTemplate call — or from waiting on its returned operation — while updating an existing InstanceGroupManager's instance template during `kops update cluster`. The underlying Google API error (wrapped via %v) carries the real cause. It is only reached when the MIG exists and the diff shows a changed InstanceTemplate.

Source

Thrown at upup/pkg/fi/cloudup/gcetasks/instancegroupmanager.go:153

		}
	} else {
		if changes.TargetPools != nil {
			op, err := t.Cloud.Compute().InstanceGroupManagers().SetTargetPools(t.Cloud.Project(), *e.Zone, i.Name, i.TargetPools)
			if err != nil {
				return fmt.Errorf("error updating TargetPools for InstanceGroupManager: %v", err)
			}

			if err := t.Cloud.WaitForOp(op); err != nil {
				return fmt.Errorf("error updating TargetPools for InstanceGroupManager: %v", err)
			}

			changes.TargetPools = nil
		}

		if changes.InstanceTemplate != nil {
			op, err := t.Cloud.Compute().InstanceGroupManagers().SetInstanceTemplate(t.Cloud.Project(), *e.Zone, i.Name, instanceTemplateURL)
			if err != nil {
				return fmt.Errorf("error updating InstanceTemplate for InstanceGroupManager: %v", err)
			}

			if err := t.Cloud.WaitForOp(op); err != nil {
				return fmt.Errorf("error updating InstanceTemplate for InstanceGroupManager: %v", err)
			}

			changes.InstanceTemplate = nil
		}

		if changes.TargetSize != nil {
			newSize := int64(0)
			if i.TargetSize != 0 {
				newSize = int64(i.TargetSize)
			}
			op, err := t.Cloud.Compute().InstanceGroupManagers().Resize(t.Cloud.Project(), *e.Zone, i.Name, newSize)
			if err != nil {
				return fmt.Errorf("error resizing InstanceGroupManager: %v", err)
			}

View on GitHub (pinned to 4c8573c808)

Solutions

1. Run with -v=10 and inspect the wrapped Google API error / operation error for the concrete cause.
2. Confirm the instance template exists: `gcloud compute instance-templates list` and that e.InstanceTemplate.URL(project) points at it.
3. Validate the template's machineType/image are available in the MIG's zone; fix the spec and re-run `kops update cluster`.
4. Check quota in the region (`gcloud compute project-info describe`) if the op fails while creating instances.
5. Wait for any in-flight MIG operation to finish, then retry.

Example fix

// before: template references unavailable machine type
//   machineType: n2-standard-8  (not offered in us-west1-a)
// after: pick an available type
// kops edit cluster  # set machineType: n2-standard-4
// kops update cluster <name> --yes

// Verify the instance template resolvable by kOps exists before updating
tmplURL, err := igm.InstanceTemplate.URL(project)
if err != nil {
    return err
}
if _, err := computeService.InstanceTemplates.Get(project, lastPathComponent(tmplURL)).Do(); err != nil {
    return fmt.Errorf("instance template %s not found: %w", tmplURL, err)
}

func isGceNotFound(err error) bool {
    var apiErr *googleapi.Error
    return errors.As(err, &apiErr) && apiErr.Code == 404
}

err := kopsUpdateCluster(...)
var apiErr *googleapi.Error
if errors.As(err, &apiErr) {
    if apiErr.Code == 404 {
        // instance template missing: recreate via kops, then re-run update
    } else if apiErr.Code == 409 {
        // operation conflict: wait for in-flight op, retry
    }
}

When it happens

Trigger: 1) SetInstanceTemplate returns an API error: instanceTemplateURL invalid/unresolvable, template deleted, zone mismatch, permission denied, or MIG busy. 2) The call succeeds but Cloud.WaitForOp(op) fails because the rolling replacement of instances hit an error (e.g. new template's image/machineType is invalid, quota exceeded while creating instances).

Common situations: Upgrading Kubernetes version where the referenced InstanceTemplate was modified or deleted out-of-band; using a machine type or image not available in the zone; GCP quota exhausted when new instances launch; a concurrent MIG operation blocking the update.

Related errors

• error resizing InstanceGroupManager: %v
• instance %s did not have Version set
• ig name not set on instance template %s
• error deleting InstanceGroupManager %s: %w
• error resizing InstanceGroupManager %s to 0: %w

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/4a96d39a2adf3e39. Report an issue: GitHub.