k3s-io/k3s · error

invalid hash: %s found on node %s

Error message

invalid hash: %s found on node %s

What it means

After stage agreement, verifyEncryptionHashAnnotation recomputes the sha256 of the local encryption configuration (GenEncryptionConfigHash) and compares it with the hash stored in the node annotation. A mismatch means the on-disk encryption-config.yaml is not the file the annotation was written for - the config drifted from the recorded state.

Source

Thrown at pkg/server/handlers/secrets-encrypt.go:595

	}

	if prevStage == "" {
		return nil
	}

	oldStage, oldHash, err := getEncryptionHashAnnotation(core)
	if err != nil {
		return err
	}

	encryptionConfigHash, err := secretsencrypt.GenEncryptionConfigHash(runtime)
	if err != nil {
		return err
	}
	if !strings.Contains(prevStage, oldStage) {
		return fmt.Errorf("incorrect stage: %s found on node %s", oldStage, nodes.Items[0].ObjectMeta.Name)
	} else if oldHash != encryptionConfigHash {
		return fmt.Errorf("invalid hash: %s found on node %s", oldHash, nodes.Items[0].ObjectMeta.Name)
	}

	return nil
}

View on GitHub (pinned to 6ba341e396)

Solutions

1. Make config and annotation consistent again: copy the exact encryption-config.yaml (and keys) from a control-plane node whose 'secrets-encrypt status' is healthy, then let the annotation be rewritten by re-running the current stage.
2. If no good copy exists cluster-wide, export secrets safely, remove the encryption config, and re-bootstrap encryption via enable + prepare.
3. Never edit the YAML by hand or restore the cred directory partially; restore data-dir snapshots atomically.
4. Verify with 'k3s secrets-encrypt status' that hash and stage agree on all servers before advancing.

Example fix

# restore exact config from the healthy node, then re-sync
scp healthy:/var/lib/rancher/k3s/server/cred/encryption-config.yaml /var/lib/rancher/k3s/server/cred/
systemctl restart k3s && k3s secrets-encrypt status

// Recompute config hash and compare with annotation before stage calls
h, err := secretsencrypt.GenEncryptionConfigHash(runtime)
if err != nil { log.Fatal(err) }
if h != annHash { log.Fatal("on-disk config does not match recorded hash - restore consistent state") }

When it happens

Trigger: The annotation survived but /var/lib/rancher/k3s/server/cred/encryption-config.yaml was modified, replaced with a version from another node/backup, or truncated/restored incompletely; any stage request then fails the oldHash != encryptionConfigHash check on nodes.Items[0].

Common situations: Restoring server state from backup that pairs a new config with an old annotation (or vice versa); hand-editing key order or formatting in the YAML (hash is over exact bytes); copying cred files between nodes with mismatched annotations.

Related errors

• unable to enable/disable secrets encryption, unknown configu
• more than 3 providers (%d) found in secrets encryption
• cannot enable secrets encryption with %s key type, no keys f
• etcd-s3-timeout must be greater than 0s
• invalid flag use; --cluster-reset required with --cluster-re

AI-assisted analysis of k3s-io/k3s@6ba341e396 (2026-08-15). Data as JSON: /api/errors/c2d55a3cacbbbd57. Report an issue: GitHub.