k3s-io/k3s · error

hash does not match between %s and %s

Error message

hash does not match between %s and %s

What it means

verifyEncryptionHashAnnotation lists all control-plane nodes and requires that every node carrying the encryption hash annotation reports the identical value (same config hash and stage). The first annotated node becomes the baseline; any other annotated node with a different value aborts stage verification with this error naming both nodes.

Source

Thrown at pkg/server/handlers/secrets-encrypt.go:575

// verifyEncryptionHashAnnotation checks that all nodes are on the same stage,
// and that a request for new stage is valid
func verifyEncryptionHashAnnotation(runtime *config.ControlRuntime, core core.Interface, prevStage string) error {
	var firstHash string
	var firstNodeName string
	first := true
	labelSelector := labels.Set{util.ControlPlaneRoleLabelKey: "true"}.String()
	nodes, err := core.V1().Node().List(metav1.ListOptions{LabelSelector: labelSelector})
	if err != nil {
		return err
	}
	for _, node := range nodes.Items {
		hash, ok := node.Annotations[secretsencrypt.EncryptionHashAnnotation]
		if ok && first {
			firstHash = hash
			first = false
			firstNodeName = node.ObjectMeta.Name
		} else if ok && hash != firstHash {
			return fmt.Errorf("hash does not match between %s and %s", firstNodeName, node.ObjectMeta.Name)
		}
	}

	if prevStage == "" {
		return nil
	}

	oldStage, oldHash, err := getEncryptionHashAnnotation(core)
	if err != nil {
		return err
	}

	encryptionConfigHash, err := secretsencrypt.GenEncryptionConfigHash(runtime)
	if err != nil {
		return err
	}
	if !strings.Contains(prevStage, oldStage) {
		return fmt.Errorf("incorrect stage: %s found on node %s", oldStage, nodes.Items[0].ObjectMeta.Name)

View on GitHub (pinned to 6ba341e396)

Solutions

1. Identify the divergent nodes: kubectl get nodes -o custom-columns=NAME:.metadata.name,ANN:.metadata.annotations -l node-role.kubernetes.io/control-plane.
2. Converge state: either restore that node's encryption-config.yaml and keys from the majority/current node, or let it re-sync by restarting k3s on it after copying cred files.
3. Re-run the same stage so all nodes rewrite a consistent annotation.
4. Avoid powering down servers across an encryption stage boundary; complete stages before maintenance.

Example fix

# converge out-of-sync node (run on the stale node)
systemctl stop k3s
scp current:/var/lib/rancher/k3s/server/cred/encryption-config.yaml /var/lib/rancher/k3s/server/cred/
systemctl start k3s && k3s secrets-encrypt status

// Verify all annotated control-plane nodes agree before stage requests
var first string
for _, n := range nodes.Items {
    if h, ok := n.Annotations["encryption.hash"]; ok {
        if first == "" { first = h } else if h != first {
            log.Fatalf("annotation divergence between nodes - converge first")
        }
    }
}

When it happens

Trigger: A secrets-encrypt stage request when two or more control-plane nodes hold different '<hash>-<stage>' annotations: one server was down while a stage advanced, an encryption config was restored on one node from backup, or a node rejoined with stale state.

Common situations: A powered-off server during 'secrets-encrypt prepare/rotate'; restoring /var/lib/rancher/k3s/server/cred on one node from an older snapshot; partial stage application interrupted mid-rollout.

Related errors

• invalid annotation %s found on node %s
• missing annotation on node %s
• unable to enable/disable secrets encryption, unknown configu
• method not allowed
• prepare does not support secretbox key type, use rotate-keys

AI-assisted analysis of k3s-io/k3s@6ba341e396 (2026-08-15). Data as JSON: /api/errors/0225c9066d57aae7. Report an issue: GitHub.