k3s-io/k3s · error

invalid annotation %s found on node %s

Error message

invalid annotation %s found on node %s

What it means

The node annotation used to coordinate secrets-encryption stages stores '<sha256-hash>-<stage>' (split on '-'). getEncryptionHashAnnotation validates that the value splits into exactly two parts; a value with zero, or two or more dashes fails this check and names the offending node.

Source

Thrown at pkg/server/handlers/secrets-encrypt.go:527

		keys.SBKeys = append(keys.SBKeys, newKey...)
	}
	logrus.Infoln("Adding secrets-encryption key: ", newKey)
	return nil
}

func getEncryptionHashAnnotation(core core.Interface) (string, string, error) {
	nodeName := os.Getenv("NODE_NAME")
	node, err := core.V1().Node().Get(nodeName, metav1.GetOptions{})
	if err != nil {
		return "", "", err
	}
	if _, ok := node.Labels[util.ControlPlaneRoleLabelKey]; !ok {
		return "", "", fmt.Errorf("cannot manage secrets encryption on non control-plane node %s", nodeName)
	}
	if ann, ok := node.Annotations[secretsencrypt.EncryptionHashAnnotation]; ok {
		split := strings.Split(ann, "-")
		if len(split) != 2 {
			return "", "", fmt.Errorf("invalid annotation %s found on node %s", ann, nodeName)
		}
		return split[0], split[1], nil
	}
	return "", "", fmt.Errorf("missing annotation on node %s", nodeName)
}

// verifyRotateKeysSupport checks that the k3s version is at least v1.28.0 on all control-plane nodes
func verifyRotateKeysSupport(core core.Interface) error {
	labelSelector := labels.Set{util.ControlPlaneRoleLabelKey: "true"}.String()
	nodes, err := core.V1().Node().List(metav1.ListOptions{LabelSelector: labelSelector})
	if err != nil {
		return err
	}
	for _, node := range nodes.Items {
		kubver, err := semver.ParseTolerant(node.Status.NodeInfo.KubeletVersion)
		if err != nil {
			return fmt.Errorf("failed to parse kubelet version %s: %v", node.Status.NodeInfo.KubeletVersion, err)
		}

View on GitHub (pinned to 6ba341e396)

Solutions

1. Read the current value: kubectl get node <name> -o jsonpath='{.metadata.annotations}' and inspect the encryption-hash annotation.
2. Copy the exact intact '<hash>-<stage>' value from a healthy control-plane node running the same stage, or delete the annotation cluster-wide and re-run 'secrets-encrypt prepare' to regenerate consistent state.
3. Never hand-edit the annotation; drive all stage transitions through the k3s CLI so the writer keeps the format.
4. Re-check 'k3s secrets-encrypt status' on all servers after repairing.

Example fix

# before
kubectl annotate node n1 encryption.hash="stage-prepare-abc123" --overwrite  # 3 parts

# after
kubectl annotate node n1 encryption.hash="abc123-prepare" --overwrite  # hash-stage

// Validate annotation shape before invoking stages
parts := strings.Split(ann, "-")
if len(parts) != 2 || parts[0] == "" || parts[1] == "" {
    log.Fatalf("malformed encryption hash annotation %q - repair before continuing", ann)
}

func validHashAnnotation(ann string) bool {
    p := strings.Split(ann, "-")
    return len(p) == 2 && p[0] != "" && p[1] != ""
}

When it happens

Trigger: The EncryptionHashAnnotation on the local control-plane node contains a malformed value: manually edited to plain text, a hash with the stage appended twice, or pasted with stray characters. getEncryptionHashAnnotation splits on '-' and len(split) != 2 triggers the error.

Common situations: Operators copying annotation values between nodes with kubectl annotate and truncating/duplicating part; external automation writing its own annotation format; leftovers from a failed or interrupted manual migration.

Related errors

• missing annotation on node %s
• hash does not match between %s and %s
• unable to enable/disable secrets encryption, unknown configu
• method not allowed
• prepare does not support secretbox key type, use rotate-keys

AI-assisted analysis of k3s-io/k3s@6ba341e396 (2026-08-15). Data as JSON: /api/errors/3ce607cce9ceae74. Report an issue: GitHub.