k3s-io/k3s · error

missing annotation on node %s

Error message

missing annotation on node %s

What it means

getEncryptionHashAnnotation requires the local control-plane node to carry the encryption hash annotation. If the Node object has the control-plane label but no annotation at all, the stage/hash cannot be determined and this error is returned instead of guessing.

Source

Thrown at pkg/server/handlers/secrets-encrypt.go:531

}

func getEncryptionHashAnnotation(core core.Interface) (string, string, error) {
	nodeName := os.Getenv("NODE_NAME")
	node, err := core.V1().Node().Get(nodeName, metav1.GetOptions{})
	if err != nil {
		return "", "", err
	}
	if _, ok := node.Labels[util.ControlPlaneRoleLabelKey]; !ok {
		return "", "", fmt.Errorf("cannot manage secrets encryption on non control-plane node %s", nodeName)
	}
	if ann, ok := node.Annotations[secretsencrypt.EncryptionHashAnnotation]; ok {
		split := strings.Split(ann, "-")
		if len(split) != 2 {
			return "", "", fmt.Errorf("invalid annotation %s found on node %s", ann, nodeName)
		}
		return split[0], split[1], nil
	}
	return "", "", fmt.Errorf("missing annotation on node %s", nodeName)
}

// verifyRotateKeysSupport checks that the k3s version is at least v1.28.0 on all control-plane nodes
func verifyRotateKeysSupport(core core.Interface) error {
	labelSelector := labels.Set{util.ControlPlaneRoleLabelKey: "true"}.String()
	nodes, err := core.V1().Node().List(metav1.ListOptions{LabelSelector: labelSelector})
	if err != nil {
		return err
	}
	for _, node := range nodes.Items {
		kubver, err := semver.ParseTolerant(node.Status.NodeInfo.KubeletVersion)
		if err != nil {
			return fmt.Errorf("failed to parse kubelet version %s: %v", node.Status.NodeInfo.KubeletVersion, err)
		}
		supportVer, err := semver.Make("1.28.0")
		if err != nil {
			return err
		}

View on GitHub (pinned to 6ba341e396)

Solutions

1. From a node that has the annotation, re-run the current stage ('k3s secrets-encrypt prepare' or the appropriate stage) so all control-plane nodes converge and receive the annotation.
2. If the whole cluster lost annotations, run 'secrets-encrypt prepare' fresh and complete the subsequent stages in order.
3. Check for admission controllers or automation that strips annotations and exclude the encryption-hash key.
4. Confirm the node in question actually runs a current k3s server build (annotation writing requires a version that supports staged rotation).

// Pre-check: every control-plane node has the annotation
nodes, _ := clientset.CoreV1().Nodes().List(metav1.ListOptions{LabelSelector: "node-role.kubernetes.io/control-plane=true"})
for _, n := range nodes.Items {
    if _, ok := n.Annotations["encryption.hash"]; !ok {
        log.Printf("node %s lacks annotation - re-run current stage", n.Name)
    }
}

When it happens

Trigger: Any secrets-encrypt operation (status/prepare/rotate stages) where the local Node lacks the EncryptionHashAnnotation: a server that joined after encryption was configured and never ran a stage, a node that was down during the last prepare, or the annotation was deleted.

Common situations: Adding a new server to an encrypted cluster and immediately running rotation; a node rejoining after restore without annotations; someone pruning 'unknown' annotations with a cleanup controller.

Related errors

• invalid annotation %s found on node %s
• hash does not match between %s and %s
• unable to enable/disable secrets encryption, unknown configu
• method not allowed
• prepare does not support secretbox key type, use rotate-keys

AI-assisted analysis of k3s-io/k3s@6ba341e396 (2026-08-15). Data as JSON: /api/errors/b8b58c607f0d7e47. Report an issue: GitHub.