k3s-io/k3s · error

node name annotation for node

Error message

node name annotation for node %s not found

What it means

The etcd member controller watches nodes with the etcd role label; when a node carries the removal annotation (etcd.k3s.cattle.io/remove), removal requires both etcd.k3s.cattle.io/node-name and etcd.k3s.cattle.io/node-address annotations, which are normally written by the metadata controller of a healthy etcd member. A removal request without the node-name annotation fails here and the controller will retry.

Solutions

1. Inspect annotations: `kubectl get node <n> -o yaml` - confirm etcd.k3s.cattle.io/node-name / node-address are missing while the remove annotation is present.
2. If removal is not intended, drop the request: `kubectl annotate node <n> etcd.k3s.cattle.io/remove-`.
3. If removal is intended and the node is gone, supply the missing annotations yourself from the etcd member list (`etcdctl member list` name) so the controller can proceed, then let it clean up.
4. If the node is still alive and should be a member, remove the remove annotation and restart its k3s server so the metadata controller republishes node-name/node-address.

Example fix

# before
kubectl annotate node bad-node etcd.k3s.cattle.io/remove=true
# -> node name annotation for node bad-node not found

# after (provide the metadata the controller needs, taken from etcdctl member list)
kubectl annotate node bad-node \
  etcd.k3s.cattle.io/node-name=bad-node \
  etcd.k3s.cattle.io/node-address=10.0.0.7

// Before requesting etcd node removal, require the metadata annotations:
node, _ := clientset.CoreV1().Nodes().Get(ctx, name, metav1.GetOptions{})
if _, hasRemove := node.Annotations["etcd.k3s.cattle.io/remove"]; hasRemove {
    if node.Annotations["etcd.k3s.cattle.io/node-name"] == "" ||
        node.Annotations["etcd.k3s.cattle.io/node-address"] == "" {
        log.Fatal("cannot request removal: node is missing etcd node-name/node-address annotations")
    }
}

When it happens

Trigger: `kubectl annotate node <n> etcd.k3s.cattle.io/remove=true` on a node labeled node-role.kubernetes.io/etcd=true while the etcd.k3s.cattle.io/node-name annotation is absent (pkg/etcd/member_controller.go:51-56) - the node never registered its member metadata, or annotations were stripped/edited.

Common situations: Manually requesting removal of a node that crashed before the metadata controller annotated it; annotations deleted during cleanup or by a mutating webhook; stale remove annotation from a previous migration.

Related errors

• cannot use S3 config secret when restoring snapshot…
• duplicate node name found, please use a unique name for…
• etcd datastore disabled
• etcd disabled
• etcd: snapshot path does not exist

AI-assisted analysis of k3s-io/k3s@6ba341e396 (2026-08-15). Data as JSON: /api/errors/76e52ce4927c5734. Report an issue: GitHub.

Appendix: source

Thrown at pkg/etcd/member_controller.go:55

	nodeController controllerv1.NodeController
	ctx            context.Context
}

func (e *etcdMemberHandler) sync(key string, node *v1.Node) (*v1.Node, error) {
	if node == nil {
		return nil, nil
	}

	if _, ok := node.Labels[util.ETCDRoleLabelKey]; !ok {
		logrus.Debugf("Node %s was not labeled etcd node, skipping sync", key)
		return node, nil
	}

	if removalRequested, ok := node.Annotations[removalAnnotation]; ok {
		// removal requires node name and address annotations; fail if either are not found
		name, ok := node.Annotations[NodeNameAnnotation]
		if !ok {
			return node, fmt.Errorf("node name annotation for node %s not found", key)
		}
		address, ok := node.Annotations[NodeAddressAnnotation]
		if !ok {
			return node, fmt.Errorf("node address annotation for node %s not found", key)
		}
		lf := logrus.Fields{"name": name, "address": address}

		patch := util.NewPatchList()
		patcher := util.NewPatcher[*v1.Node](e.nodeController)

		// Check to see if the node was previously removed from the cluster
		if removed, ok := node.Annotations[removedNodeNameAnnotation]; ok {
			if removed != name {
				// If the current node name is not the same as the removed node name, clear the removal annotations,
				// as this indicates that the node has been re-added with a new name.
				logrus.WithFields(lf).Info("Resetting removed node flag as removed node name does not match current node name")
				patch.Remove("metadata", "annotations", removedNodeNameAnnotation)
				patch.Remove("metadata", "annotations", removalAnnotation)

View on GitHub (pinned to 6ba341e396)