kubernetes/kops · error

error resizing InstanceGroupManager: %v

Error message

error resizing InstanceGroupManager: %v

What it means

kOps wraps a failure from the GCE Compute API Resize call on an existing InstanceGroupManager when the desired TargetSize differs from the live one. The wrapped %v value is the underlying Google API error. newSize falls back to 0 when the spec's TargetSize is 0, which itself can trigger an API validation error if the MIG cannot shrink that far.

Source

Thrown at upup/pkg/fi/cloudup/gcetasks/instancegroupmanager.go:170

			if err != nil {
				return fmt.Errorf("error updating InstanceTemplate for InstanceGroupManager: %v", err)
			}

			if err := t.Cloud.WaitForOp(op); err != nil {
				return fmt.Errorf("error updating InstanceTemplate for InstanceGroupManager: %v", err)
			}

			changes.InstanceTemplate = nil
		}

		if changes.TargetSize != nil {
			newSize := int64(0)
			if i.TargetSize != 0 {
				newSize = int64(i.TargetSize)
			}
			op, err := t.Cloud.Compute().InstanceGroupManagers().Resize(t.Cloud.Project(), *e.Zone, i.Name, newSize)
			if err != nil {
				return fmt.Errorf("error resizing InstanceGroupManager: %v", err)
			}

			if err := t.Cloud.WaitForOp(op); err != nil {
				return fmt.Errorf("error resizing InstanceGroupManager: %v", err)
			}

			changes.TargetSize = nil
		}

		empty := &InstanceGroupManager{}
		if !reflect.DeepEqual(empty, changes) {
			return fmt.Errorf("cannot apply changes to InstanceGroupManager: %v", changes)
		}
	}

	return nil
}

View on GitHub (pinned to 4c8573c808)

Solutions

1. Read the wrapped error detail with -v=10 logging to identify the concrete API/operation failure.
2. Check and raise GCE quota (CPUs, in-use IPs) for the region if growing: `gcloud compute project-info describe`.
3. Retry `kops update cluster` once any in-flight MIG operation completes.
4. If shrinking, verify no pinned/standby instances block removal; resize in smaller steps if needed.

Example fix

// before: TargetSize above regional quota
//   kops set cluster spec.nodes.maxSize=100  # exceeds 48-CPU quota
// after
//   kops set cluster spec.nodes.maxSize=24
//   kops update cluster <name> --yes

// Ensure requested size fits within regional quota before resizing
quota := getRegionalQuota(project, region, "CPUS")
projected := currentCores + int64(newTargetSize-desiredCurrent)*coresPerInstance
if projected > quota.Limit {
    return fmt.Errorf("resize to %d needs %d cores but quota is %d", newTargetSize, projected, quota.Limit)
}

func isQuotaExceeded(err error) bool {
    var apiErr *googleapi.Error
    if errors.As(err, &apiErr) && apiErr.Code == 403 {
        return strings.Contains(apiErr.Message, "Quota")
    }
    return false
}

When it happens

Trigger: 1) Resize returns an API error: quota exceeded for the new (larger) size, MIG busy with another operation (e.g. the SetInstanceTemplate op just before it), invalid target size. 2) Resize succeeds but Cloud.WaitForOp(op) fails while instances are being added/removed (creation failure, quota).

Common situations: Scaling a node ASG/MIG above remaining regional CPU quota; shrinking to 0 while instances are in a failed state; autoscaler or another kOps run concurrently resizing the group.

Related errors

• error resizing InstanceGroupManager %s to 0: %w
• error updating InstanceTemplate for InstanceGroupManager: %v
• error deleting InstanceGroupManager %s: %w
• error listing ManagedInstances in %s: %w
• cannot apply changes to InstanceGroupManager: %v

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/2179af7af3119cfc. Report an issue: GitHub.