k3s-io/k3s · error

node name annotation for node %s not found

Error message

node name annotation for node %s not found

What it means

The etcd member controller watches nodes with the etcd role label; when a node carries the removal annotation (etcd.k3s.cattle.io/remove), removal requires both etcd.k3s.cattle.io/node-name and etcd.k3s.cattle.io/node-address annotations, which are normally written by the metadata controller of a healthy etcd member. A removal request without the node-name annotation fails here and the controller will retry.

Source

Thrown at pkg/etcd/member_controller.go:55

	nodeController controllerv1.NodeController
	ctx            context.Context
}

func (e *etcdMemberHandler) sync(key string, node *v1.Node) (*v1.Node, error) {
	if node == nil {
		return nil, nil
	}

	if _, ok := node.Labels[util.ETCDRoleLabelKey]; !ok {
		logrus.Debugf("Node %s was not labeled etcd node, skipping sync", key)
		return node, nil
	}

	if removalRequested, ok := node.Annotations[removalAnnotation]; ok {
		// removal requires node name and address annotations; fail if either are not found
		name, ok := node.Annotations[NodeNameAnnotation]
		if !ok {
			return node, fmt.Errorf("node name annotation for node %s not found", key)
		}
		address, ok := node.Annotations[NodeAddressAnnotation]
		if !ok {
			return node, fmt.Errorf("node address annotation for node %s not found", key)
		}
		lf := logrus.Fields{"name": name, "address": address}

		patch := util.NewPatchList()
		patcher := util.NewPatcher[*v1.Node](e.nodeController)

		// Check to see if the node was previously removed from the cluster
		if removed, ok := node.Annotations[removedNodeNameAnnotation]; ok {
			if removed != name {
				// If the current node name is not the same as the removed node name, clear the removal annotations,
				// as this indicates that the node has been re-added with a new name.
				logrus.WithFields(lf).Info("Resetting removed node flag as removed node name does not match current node name")
				patch.Remove("metadata", "annotations", removedNodeNameAnnotation)
				patch.Remove("metadata", "annotations", removalAnnotation)

View on GitHub (pinned to 6ba341e396)

Solutions

1. Inspect annotations: `kubectl get node <n> -o yaml` - confirm etcd.k3s.cattle.io/node-name / node-address are missing while the remove annotation is present.
2. If removal is not intended, drop the request: `kubectl annotate node <n> etcd.k3s.cattle.io/remove-`.
3. If removal is intended and the node is gone, supply the missing annotations yourself from the etcd member list (`etcdctl member list` name) so the controller can proceed, then let it clean up.
4. If the node is still alive and should be a member, remove the remove annotation and restart its k3s server so the metadata controller republishes node-name/node-address.

Example fix

# before
kubectl annotate node bad-node etcd.k3s.cattle.io/remove=true
# -> node name annotation for node bad-node not found

# after (provide the metadata the controller needs, taken from etcdctl member list)
kubectl annotate node bad-node \
  etcd.k3s.cattle.io/node-name=bad-node \
  etcd.k3s.cattle.io/node-address=10.0.0.7

// Before requesting etcd node removal, require the metadata annotations:
node, _ := clientset.CoreV1().Nodes().Get(ctx, name, metav1.GetOptions{})
if _, hasRemove := node.Annotations["etcd.k3s.cattle.io/remove"]; hasRemove {
    if node.Annotations["etcd.k3s.cattle.io/node-name"] == "" ||
        node.Annotations["etcd.k3s.cattle.io/node-address"] == "" {
        log.Fatal("cannot request removal: node is missing etcd node-name/node-address annotations")
    }
}

When it happens

Trigger: `kubectl annotate node <n> etcd.k3s.cattle.io/remove=true` on a node labeled node-role.kubernetes.io/etcd=true while the etcd.k3s.cattle.io/node-name annotation is absent (pkg/etcd/member_controller.go:51-56) - the node never registered its member metadata, or annotations were stripped/edited.

Common situations: Manually requesting removal of a node that crashed before the metadata controller annotated it; annotations deleted during cleanup or by a mutating webhook; stale remove annotation from a previous migration.

Related errors

• no bootstrap data found in datastore - check server token va
• etcd datastore disabled
• no bootstrap data found
• found multiple bootstrap keys in storage
• this server has not yet been promoted from learner to voting

AI-assisted analysis of k3s-io/k3s@6ba341e396 (2026-08-15). Data as JSON: /api/errors/76e52ce4927c5734. Report an issue: GitHub.