kubernetes/kops · error

waiting for VMSS deletion completion: %w

Error message

waiting for VMSS deletion completion: %w

What it means

This error wraps a failure reported by the ARM async long-running-operation poller (future.PollUntilDone) after the delete request was accepted. The VMSS deletion either failed inside Azure (e.g. dependent resources, provisioning error) or polling itself hit timeouts/HTTP errors. The wrapped error carries the poller's terminal state or the intermediate HTTP failure.

Source

Thrown at upup/pkg/fi/cloudup/azure/vmscaleset.go:94

func (c *vmScaleSetsClientImpl) Get(ctx context.Context, resourceGroupName string, vmssName string) (*compute.VirtualMachineScaleSet, error) {
	opts := &compute.VirtualMachineScaleSetsClientGetOptions{
		Expand: to.Ptr(compute.ExpandTypesForGetVMScaleSetsUserData),
	}
	resp, err := c.c.Get(ctx, resourceGroupName, vmssName, opts)
	if err != nil {
		return nil, fmt.Errorf("getting VMSS: %w", err)
	}
	return &resp.VirtualMachineScaleSet, nil
}

func (c *vmScaleSetsClientImpl) Delete(ctx context.Context, resourceGroupName, vmssName string) error {
	future, err := c.c.BeginDelete(ctx, resourceGroupName, vmssName, nil)
	if err != nil {
		return fmt.Errorf("deleting VMSS: %w", err)
	}
	if _, err := future.PollUntilDone(ctx, nil); err != nil {
		return fmt.Errorf("waiting for VMSS deletion completion: %w", err)
	}
	return nil
}

func newVMScaleSetsClientImpl(subscriptionID string, cred *azidentity.DefaultAzureCredential) (*vmScaleSetsClientImpl, error) {
	c, err := compute.NewVirtualMachineScaleSetsClient(subscriptionID, cred, nil)
	if err != nil {
		return nil, fmt.Errorf("creating VMSSs client: %w", err)
	}
	return &vmScaleSetsClientImpl{
		c: c,
	}, nil
}

View on GitHub (pinned to 4c8573c808)

Solutions

1. Increase the context deadline passed to Delete (long LROs can take 10-30+ min for large scale sets)
2. Check for Azure resource locks (az lock list) on the resource group or scale set and remove conflicting locks
3. Inspect the failed LRO details: unwrap and read the ResponseError to find the ARM error code for the underlying instance delete failure
4. Retry the whole Delete; a partially-deleted VMSS can be re-deleted idempotently
5. Verify no cancellation source (kops abort/CI timeout) is killing ctx mid-poll

Example fix

// before
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Minute)
defer cancel()
err := client.Delete(ctx, rg, vmssName)
// after
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Minute) // LRO can be slow
defer cancel()
err := client.Delete(ctx, rg, vmssName)

// Go: use a generous deadline for the long-running delete and check locks first
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Minute)
defer cancel()
// pre-check for Azure locks that would fail the LRO:
// locks := locksClient.ListByResourceGroup(ctx, rg, nil) // abort if a Delete lock exists

func isLRORetries(err error) bool {
	var respErr *azcore.ResponseError
	return errors.As(err, &respErr) && (respErr.StatusCode == 429 || respErr.StatusCode >= 500 || errors.Is(err, context.DeadlineExceeded))
}

err := client.Delete(ctx, rg, vmssName)
if err != nil {
	if isLRORetries(err) {
		// re-issue Delete with backoff; LROs are safely re-runnable
		return retryWithBackoff(ctx, func() error { return client.Delete(ctx, rg, vmssName) })
	}
	return fmt.Errorf("waiting for VMSS deletion completion: %w", err)
}

When it happens

Trigger: BeginDelete returned a future successfully, then PollUntilDone observes: the LRO reached a Failed state (Azure could not delete underlying NICs/disks/VM instances), HTTP polling errors (401 token expiry mid-operation, 429 throttling of the polling GET), or the ctx passed to PollUntilDone was cancelled/timed out before completion.

Common situations: Long VMSS deletions exceeding the caller's context timeout (kops delete cluster run under a CI timeout); Azure region outage or capacity issues blocking instance deletion; deletion blocked by resources locked (ReadOnly/Delete locks) in the portal; identity token expiring during very long polls.

Related errors

• waiting for disk create/update completion: %w
• waiting for public ip address deletion completion: %w
• waiting for virtual network deletion completion: %w
• waiting for VMSS create/update: %w
• deleting VMSS: %w

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/81df433222ac45e0. Report an issue: GitHub.