kubernetes/kops · error

error listing machines: %w

Error message

error listing machines: %w

What it means

In the Cluster API nodeidentity manager, FindMachineByProviderID lists all cluster.x-k8s.io/v1beta1 Machine objects via the controller-runtime client. If the List call against the management cluster fails, the error is wrapped as "error listing machines". The root cause is the wrapped client error: RBAC denial, unreachable API server, or CRD/mismatched API version.

Source

Thrown at pkg/nodeidentity/clusterapi/capimanager/manager.go:54

func NewManager(kubeClient client.Client) *Manager {
	return &Manager{
		kubeClient: kubeClient,
	}
}

func (m *Manager) FindMachineByProviderID(ctx context.Context, providerID string) (*clusterapi.Machine, error) {
	// TODO: Can we build an index
	// selector := client.MatchingFieldsSelector{
	// 	Selector: fields.OneTermEqualSelector("spec.providerID", providerID),
	// }
	var machines unstructured.UnstructuredList
	machines.SetGroupVersionKind(schema.GroupVersionKind{
		Group:   "cluster.x-k8s.io",
		Kind:    "Machine",
		Version: "v1beta1",
	})
	if err := m.kubeClient.List(ctx, &machines); err != nil {
		return nil, fmt.Errorf("error listing machines: %w", err)
	}
	var matches []*unstructured.Unstructured
	for i := range machines.Items {
		machine := &machines.Items[i]
		machineSpecProviderID, _, _ := unstructured.NestedString(machine.Object, "spec", "providerID")
		if machineSpecProviderID != providerID {
			continue
		}
		matches = append(matches, machine)
	}
	if len(matches) > 0 {
		if len(matches) > 1 {
			return nil, fmt.Errorf("found multiple machines with providerID %q", providerID)
		}
		machine := matches[0]
		machine = machine.DeepCopy()
		return clusterapi.NewMachine(machine), nil
	}

View on GitHub (pinned to 4c8573c808)

Solutions

1. Read the wrapped error: for 'forbidden', add RBAC (list/get machines.cluster.x-k8s.io) to the caller's ServiceAccount.
2. Verify Cluster API CRDs v1beta1 exist in the management cluster (kubectl get crd machines.cluster.x-k8s.io).
3. Confirm kubeconfig/connectivity from the component to the management cluster API server.
4. Retry on transient connection errors; check API server health and rate limits.

Example fix

// before: ServiceAccount with no CAPI permissions
kind: ClusterRole
rules: []

// after
kind: ClusterRole
rules:
- apiGroups: ["cluster.x-k8s.io"]
  resources: ["machines"]
  verbs: ["get", "list", "watch"]

if err := kubeClient.List(ctx, &client.ListOptions{Limit: 1}, &unstructured.UnstructuredList{}); err != nil {
    return fmt.Errorf("management cluster unreachable / RBAC denied: %w", err)
}

machine, err := mgr.FindMachineByProviderID(ctx, providerID)
if err != nil && strings.Contains(err.Error(), "error listing machines") {
    if apierrors.IsForbidden(err) {
        return fmt.Errorf("grant list on machines.cluster.x-k8s.io to the ServiceAccount: %w", err)
    }
    return retryWithBackoff(3, func() error {
        machine, err = mgr.FindMachineByProviderID(ctx, providerID)
        return err
    })
}

When it happens

Trigger: Calling FindMachineByProviderID (via IdentifyNode or VerifyToken) when the management cluster API server is unreachable, the caller's ServiceAccount lacks list/get on cluster.x-k8s.io/machines, or the Cluster API CRDs (v1beta1) are not installed.

Common situations: Running the node identity service outside the management cluster with stale kubeconfig; missing RBAC rules in the CAPI provider's ClusterRole; upgrading Cluster API from v1alpha3/v1alpha4 to v1beta1 while CRDs are still old; network policy blocking egress to the API server.

Related errors

• found multiple machines with providerID %q
• error listing nodes: %v
• error listing Pods: %v
• error reading Keyset %q: %v
• error adding needs-update label: %v

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/272caa45b19892cd. Report an issue: GitHub.