{"record":{"id":"c99fff8a691d8270","repo":"k3s-io/k3s","slug":"failed-to-parse-kubelet-version-s-v","errorCode":null,"errorMessage":"failed to parse kubelet version %s: %v","messagePattern":"failed to parse kubelet version (.+?): (.+?)","errorType":"http","errorClass":null,"httpStatus":400,"severity":"error","filePath":"pkg/server/handlers/secrets-encrypt.go","lineNumber":544,"sourceCode":"\t\tif len(split) != 2 {\n\t\t\treturn \"\", \"\", fmt.Errorf(\"invalid annotation %s found on node %s\", ann, nodeName)\n\t\t}\n\t\treturn split[0], split[1], nil\n\t}\n\treturn \"\", \"\", fmt.Errorf(\"missing annotation on node %s\", nodeName)\n}\n\n// verifyRotateKeysSupport checks that the k3s version is at least v1.28.0 on all control-plane nodes\nfunc verifyRotateKeysSupport(core core.Interface) error {\n\tlabelSelector := labels.Set{util.ControlPlaneRoleLabelKey: \"true\"}.String()\n\tnodes, err := core.V1().Node().List(metav1.ListOptions{LabelSelector: labelSelector})\n\tif err != nil {\n\t\treturn err\n\t}\n\tfor _, node := range nodes.Items {\n\t\tkubver, err := semver.ParseTolerant(node.Status.NodeInfo.KubeletVersion)\n\t\tif err != nil {\n\t\t\treturn fmt.Errorf(\"failed to parse kubelet version %s: %v\", node.Status.NodeInfo.KubeletVersion, err)\n\t\t}\n\t\tsupportVer, err := semver.Make(\"1.28.0\")\n\t\tif err != nil {\n\t\t\treturn err\n\t\t}\n\t\tif kubver.LT(supportVer) {\n\t\t\treturn fmt.Errorf(\"node %s is running k3s version %s that does not support rotate-keys\", node.ObjectMeta.Name, kubver.String())\n\t\t}\n\t}\n\treturn nil\n}\n\n// verifyEncryptionHashAnnotation checks that all nodes are on the same stage,\n// and that a request for new stage is valid\nfunc verifyEncryptionHashAnnotation(runtime *config.ControlRuntime, core core.Interface, prevStage string) error {\n\tvar firstHash string\n\tvar firstNodeName string\n\tfirst := true","sourceCodeStart":526,"sourceCodeEnd":562,"githubUrl":"https://github.com/k3s-io/k3s/blob/6ba341e396edc16b8dcae978a7c5e3ac7ee5606e/pkg/server/handlers/secrets-encrypt.go#L526-L562","documentation":"verifyRotateKeysSupport lists all control-plane nodes and parses each node's Status.NodeInfo.KubeletVersion with semver.ParseTolerant before checking against 1.28.0. If a node reports a version string that is not even tolerable semver (empty, 'unknown', or arbitrary text), parsing fails and this wrapped error names the offending value.","triggerScenarios":"Invoking 'k3s secrets-encrypt rotate-keys' while any control-plane node's KubeletVersion field is empty or non-semver: a node still registering (kubelet has not fully populated status), a broken/custom kubelet build, or node status partially populated after a crash.","commonSituations":"Running rotate-keys immediately after booting new servers before kubelet status settles; clusters where kubelet version reporting was altered; stale NotReady nodes with garbage status retained.","solutions":["Check the reported values: kubectl get nodes -o custom-columns=NAME:.metadata.name,VER:.status.nodeInfo.kubeletVersion and find the empty/garbage entry.","Let in-progress nodes finish registering (wait for Ready) or delete stale NotReady node objects, then retry rotate-keys.","Restart k3s on the node whose version is blank so kubelet repopulates status.","Retry rotate-keys only after every control-plane node reports a valid version."],"exampleFix":"# before: node reports empty version\nNAME   VER\nn1     v1.28.4+k3s3\nn2                # <- rotate-keys fails here\n\n# after: restart k3s on n2, wait for Ready, retry\nk3s secrets-encrypt rotate-keys","handlingStrategy":"validation","validationCode":"// Pre-check all control-plane kubelet versions parse\nfor _, n := range nodes.Items {\n    if _, err := semver.ParseTolerant(n.Status.NodeInfo.KubeletVersion); err != nil {\n        log.Fatalf(\"node %s reports unparseable version %q\", n.Name, n.Status.NodeInfo.KubeletVersion)\n    }\n}","typeGuard":"func parseableVersion(v string) bool {\n    if v == \"\" { return false }\n    _, err := semver.ParseTolerant(v)\n    return err == nil\n}","tryCatchPattern":null,"preventionTips":["Wait for nodes to reach Ready before rotation operations","Delete stale NotReady node objects that report empty versions","Restart k3s on nodes with blank kubelet version fields"],"tags":["versioning","nodes","secrets-encryption"],"backgroundTag":null,"analyzedSha":"6ba341e396edc16b8dcae978a7c5e3ac7ee5606e","analyzedAt":"2026-08-15T16:27:54.286Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}