k3s-io/k3s · error

failed to parse kubelet version %s: %v

Error message

failed to parse kubelet version %s: %v

What it means

verifyRotateKeysSupport lists all control-plane nodes and parses each node's Status.NodeInfo.KubeletVersion with semver.ParseTolerant before checking against 1.28.0. If a node reports a version string that is not even tolerable semver (empty, 'unknown', or arbitrary text), parsing fails and this wrapped error names the offending value.

Source

Thrown at pkg/server/handlers/secrets-encrypt.go:544

		if len(split) != 2 {
			return "", "", fmt.Errorf("invalid annotation %s found on node %s", ann, nodeName)
		}
		return split[0], split[1], nil
	}
	return "", "", fmt.Errorf("missing annotation on node %s", nodeName)
}

// verifyRotateKeysSupport checks that the k3s version is at least v1.28.0 on all control-plane nodes
func verifyRotateKeysSupport(core core.Interface) error {
	labelSelector := labels.Set{util.ControlPlaneRoleLabelKey: "true"}.String()
	nodes, err := core.V1().Node().List(metav1.ListOptions{LabelSelector: labelSelector})
	if err != nil {
		return err
	}
	for _, node := range nodes.Items {
		kubver, err := semver.ParseTolerant(node.Status.NodeInfo.KubeletVersion)
		if err != nil {
			return fmt.Errorf("failed to parse kubelet version %s: %v", node.Status.NodeInfo.KubeletVersion, err)
		}
		supportVer, err := semver.Make("1.28.0")
		if err != nil {
			return err
		}
		if kubver.LT(supportVer) {
			return fmt.Errorf("node %s is running k3s version %s that does not support rotate-keys", node.ObjectMeta.Name, kubver.String())
		}
	}
	return nil
}

// verifyEncryptionHashAnnotation checks that all nodes are on the same stage,
// and that a request for new stage is valid
func verifyEncryptionHashAnnotation(runtime *config.ControlRuntime, core core.Interface, prevStage string) error {
	var firstHash string
	var firstNodeName string
	first := true

View on GitHub (pinned to 6ba341e396)

Solutions

1. Check the reported values: kubectl get nodes -o custom-columns=NAME:.metadata.name,VER:.status.nodeInfo.kubeletVersion and find the empty/garbage entry.
2. Let in-progress nodes finish registering (wait for Ready) or delete stale NotReady node objects, then retry rotate-keys.
3. Restart k3s on the node whose version is blank so kubelet repopulates status.
4. Retry rotate-keys only after every control-plane node reports a valid version.

Example fix

# before: node reports empty version
NAME   VER
n1     v1.28.4+k3s3
n2                # <- rotate-keys fails here

# after: restart k3s on n2, wait for Ready, retry
k3s secrets-encrypt rotate-keys

// Pre-check all control-plane kubelet versions parse
for _, n := range nodes.Items {
    if _, err := semver.ParseTolerant(n.Status.NodeInfo.KubeletVersion); err != nil {
        log.Fatalf("node %s reports unparseable version %q", n.Name, n.Status.NodeInfo.KubeletVersion)
    }
}

func parseableVersion(v string) bool {
    if v == "" { return false }
    _, err := semver.ParseTolerant(v)
    return err == nil
}

When it happens

Trigger: Invoking 'k3s secrets-encrypt rotate-keys' while any control-plane node's KubeletVersion field is empty or non-semver: a node still registering (kubelet has not fully populated status), a broken/custom kubelet build, or node status partially populated after a crash.

Common situations: Running rotate-keys immediately after booting new servers before kubelet status settles; clusters where kubelet version reporting was altered; stale NotReady nodes with garbage status retained.

Understand the failure class

• Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

• cannot manage secrets encryption on non control-plane node %
• node %s is running k3s version %s that does not support rota
• unable to enable/disable secrets encryption, unknown configu
• method not allowed
• prepare does not support secretbox key type, use rotate-keys

AI-assisted analysis of k3s-io/k3s@6ba341e396 (2026-08-15). Data as JSON: /api/errors/c99fff8a691d8270. Report an issue: GitHub.