k3s-io/k3s · error

incorrect stage: %s found on node %s

Error message

incorrect stage: %s found on node %s

What it means

verifyEncryptionHashAnnotation checks that the stage recorded on the local node is compatible with the stage transition being requested (prevStage must contain the locally found oldStage). If the cluster's nodes are not on the stage the request assumes, the transition is rejected with the found stage and node name - stages must be executed in order: prepare -> rotate -> ... -> re-encrypt-active.

Source

Thrown at pkg/server/handlers/secrets-encrypt.go:593

			return fmt.Errorf("hash does not match between %s and %s", firstNodeName, node.ObjectMeta.Name)
		}
	}

	if prevStage == "" {
		return nil
	}

	oldStage, oldHash, err := getEncryptionHashAnnotation(core)
	if err != nil {
		return err
	}

	encryptionConfigHash, err := secretsencrypt.GenEncryptionConfigHash(runtime)
	if err != nil {
		return err
	}
	if !strings.Contains(prevStage, oldStage) {
		return fmt.Errorf("incorrect stage: %s found on node %s", oldStage, nodes.Items[0].ObjectMeta.Name)
	} else if oldHash != encryptionConfigHash {
		return fmt.Errorf("invalid hash: %s found on node %s", oldHash, nodes.Items[0].ObjectMeta.Name)
	}

	return nil
}

View on GitHub (pinned to 6ba341e396)

Solutions

1. Check actual state first: 'k3s secrets-encrypt status' on every control-plane node.
2. Run the stages strictly in order (prepare, then rotate per iteration, finally re-encrypt-active) matching what status reports.
3. If the stage annotation is stale on some nodes, re-run the currently reported stage to completion before advancing.
4. Do not skip or parallelize stages with scripts; execute them sequentially and verify status between steps.

Example fix

# before
k3s secrets-encrypt re-encrypt  # nodes still in 'prepare'

# after
k3s secrets-encrypt rotate       # complete current stage first
k3s secrets-encrypt re-encrypt

// Read current stage, then only allow the documented successor
cur := stageFromAnnotation(ann)
next := map[string][]string{"prepare":{"rotate"},"rotate":{"re-encrypt-active"}}
if !contains(next[cur], requested) { log.Fatal("execute stages in order") }

func isValidTransition(from, to string) bool {
    if from == "" { return to == "prepare" }
    switch to {
    case "rotate", "re-encrypt-active":
        return from == "prepare" || from == "rotate"
    }
    return false
}

When it happens

Trigger: Requesting 'rotate' or 're-encrypt-active' when nodes still report 'prepare'; skipping a stage; calling a stage twice with the second invocation expecting the completed first one; version/tooling drift where the client and nodes disagree on the current stage.

Common situations: Operators jumping straight to re-encrypt after adding a key without completing intermediate stages; a stage that silently failed earlier on some nodes; retrying an interrupted operation out of order.

Related errors

• unknown stage %s requested
• Failed checking netMode
• Initial server URL host is not defined for load balancer
• --server is required
• service %s is not recognized

AI-assisted analysis of k3s-io/k3s@6ba341e396 (2026-08-15). Data as JSON: /api/errors/f4cb0fc928893443. Report an issue: GitHub.