kubernetes/kops · error

listing VMSS VMs: %w

Error message

listing VMSS VMs: %w

What it means

This error wraps a failure from paginating the ARM VirtualMachineScaleSetVMs List API, which enumerates the VM instances inside a scale set. Any page fetch failure (auth, throttling, VMSS vanished mid-enumeration, network) is wrapped once per pager.NextPage call with the accumulated results discarded. The SDK error is preserved via %w for errors.Is/As inspection.

Source

Thrown at upup/pkg/fi/cloudup/azure/vmscalesetvm.go:45

// VMScaleSetVMsClient is a client for managing VMs in VM Scale Sets.
type VMScaleSetVMsClient interface {
	List(ctx context.Context, resourceGroupName, vmssName string) ([]*compute.VirtualMachineScaleSetVM, error)
	Delete(ctx context.Context, resourceGroupName, vmssName, instanceId string) error
}

type vmScaleSetVMsClientImpl struct {
	c *compute.VirtualMachineScaleSetVMsClient
}

var _ VMScaleSetVMsClient = (*vmScaleSetVMsClientImpl)(nil)

func (c *vmScaleSetVMsClientImpl) List(ctx context.Context, resourceGroupName, vmssName string) ([]*compute.VirtualMachineScaleSetVM, error) {
	var l []*compute.VirtualMachineScaleSetVM
	pager := c.c.NewListPager(resourceGroupName, vmssName, nil)
	for pager.More() {
		resp, err := pager.NextPage(ctx)
		if err != nil {
			return nil, fmt.Errorf("listing VMSS VMs: %w", err)
		}
		l = append(l, resp.Value...)
	}
	return l, nil
}

func (c *vmScaleSetVMsClientImpl) Delete(ctx context.Context, resourceGroupName, vmssName, instanceId string) error {
	future, err := c.c.BeginDelete(ctx, resourceGroupName, vmssName, instanceId, nil)
	if err != nil {
		return fmt.Errorf("deleting VMSS VM: %w", err)
	}
	if _, err = future.PollUntilDone(ctx, nil); err != nil {
		return fmt.Errorf("waiting for VMSS VM deletion completion: %w", err)
	}
	return nil
}

func newVMScaleSetVMsClientImpl(subscriptionID string, cred *azidentity.DefaultAzureCredential) (*vmScaleSetVMsClientImpl, error) {

View on GitHub (pinned to 4c8573c808)

Solutions

1. Unwrap with errors.As (*azcore.ResponseError): retry on 429 honoring Retry-After, return not-found on 404
2. Reduce list frequency or scope to stay under ARM throttling limits (batch instance operations)
3. Re-check RBAC: the identity needs Reader/Virtual Machine Contributor on the resource group
4. Retry the full List — results are discarded on partial failure, so pagination must restart
5. Ensure the parent VMSS still exists (or handle 404 as an empty list) before enumerating instances

Example fix

// before
vms, err := client.List(ctx, rg, vmssName)
if err != nil { return err }
// after
vms, err := client.List(ctx, rg, vmssName)
if err != nil {
	var respErr *azcore.ResponseError
	if errors.As(err, &respErr) && respErr.StatusCode == http.StatusNotFound {
		return nil // VMSS gone: zero instances
	}
	return fmt.Errorf("listing VMSS %q instances: %w", vmssName, err)
}

// Go: verify the parent VMSS exists before enumerating its instances
_, err := scaleSetsClient.Get(ctx, rg, vmssName)
if err != nil {
	if isNotFoundErr(err) { return nil, nil } // no VMSS => zero VMs
	return nil, err
}

func isThrottledOrNotFound(err error) (throttled, notFound bool) {
	var respErr *azcore.ResponseError
	if !errors.As(err, &respErr) {
		return false, false
	}
	return respErr.StatusCode == http.StatusTooManyRequests, respErr.StatusCode == http.StatusNotFound
}

vms, err := client.List(ctx, rg, vmssName)
if err != nil {
	throttled, notFound := isThrottledOrNotFound(err)
	switch {
	case notFound:
		return nil // VMSS gone: treat as empty instance list
	case throttled:
		// honor Retry-After and re-run the full pagination
	default:
		return fmt.Errorf("listing VMSS VMs: %w", err)
	}
}

When it happens

Trigger: vmScaleSetVMsClientImpl.List iterates c.c.NewListPager(...).NextPage(ctx) and a page request fails: RBAC lacks read access to the VMSS instances, ARM returns 429 under heavy listing, the parent VMSS is deleted while paging, or a transient network failure occurs between pages.

Common situations: kops validating or updating instances of a large scale set and hitting ARM throttling limits (many pages); the scale set was resized/deleted concurrently by another controller mid-list; identity credentials lack 'Reader' on the VM; expired token during long paginations.

Related errors

• listing public ip addresses: %w
• listing resource groups: %w
• getting VMSS: %w
• deleting VMSS: %w
• expected exactly one subnet for InstanceGroup %q; subnets wa

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/6db4f8f7cf8f060c. Report an issue: GitHub.