{"record":{"id":"da8db3c94cf3bda1","repo":"tailscale/tailscale","slug":"error-writing-capability-version-to-state-store","errorCode":null,"errorMessage":"error writing capability version to state store: %w","messagePattern":"error writing capability version to state store: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"kube/state/state.go","lineNumber":39,"sourceCode":"\t\"tailscale.com/tailcfg\"\n\t\"tailscale.com/util/deephash\"\n)\n\nconst (\n\tkeyPodUID     = ipn.StateKey(kubetypes.KeyPodUID)\n\tkeyCapVer     = ipn.StateKey(kubetypes.KeyCapVer)\n\tkeyDeviceID   = ipn.StateKey(kubetypes.KeyDeviceID)\n\tkeyDeviceIPs  = ipn.StateKey(kubetypes.KeyDeviceIPs)\n\tkeyDeviceFQDN = ipn.StateKey(kubetypes.KeyDeviceFQDN)\n)\n\n// SetInitialKeys sets Pod UID and cap ver.\nfunc SetInitialKeys(store ipn.StateStore, podUID string) error {\n\tif err := store.WriteState(keyPodUID, []byte(podUID)); err != nil {\n\t\treturn fmt.Errorf(\"error writing pod UID to state store: %w\", err)\n\t}\n\tif err := store.WriteState(keyCapVer, fmt.Appendf(nil, \"%d\", tailcfg.CurrentCapabilityVersion)); err != nil {\n\t\treturn fmt.Errorf(\"error writing capability version to state store: %w\", err)\n\t}\n\n\treturn nil\n}\n\n// KeepKeysUpdated sets state store keys consistent with containerboot to\n// signal proxy readiness to the operator. It runs until its context is\n// cancelled or it hits an error. It watches the IPN bus for SelfChange\n// notifications (which fire whenever the self node changes) and reads\n// the new self node directly from the notify.\nfunc KeepKeysUpdated(ctx context.Context, store ipn.StateStore, lc klc.LocalClient) error {\n\tw, err := lc.WatchIPNBus(ctx, ipn.NotifyInitialNetMap)\n\tif err != nil {\n\t\treturn fmt.Errorf(\"error watching IPN bus: %w\", err)\n\t}\n\tdefer w.Close()\n\n\tvar currentDeviceID, currentDeviceIPs, currentDeviceFQDN deephash.Sum","sourceCodeStart":21,"sourceCodeEnd":57,"githubUrl":"https://github.com/tailscale/tailscale/blob/6e0912f97994f927632b34ae9e63b53d6516a6ac/kube/state/state.go#L21-L57","documentation":"Thrown by SetInitialKeys in tailscale.com/kube/state: after the pod-UID key was written, writing the capability-version key ('cap-ver', tailcfg.CurrentCapabilityVersion) via store.WriteState failed. In operator-managed Kubernetes deployments the store is ipn/store/kubestore.Store, so this is a Secret update against the API server. The wrapped error carries the exact API failure (status code, RBAC reason, or transport error).","triggerScenarios":"Calling state.SetInitialKeys(store, podUID) when the kubestore Secret update fails: RBAC lacking update/patch on secrets, the node's state Secret deleted mid-run, API server unreachable/429/timeout, or an admission webhook rejecting the write. The preceding pod-UID write succeeded, so the failure raced in between the two writes.","commonSituations":"Tailscale Kubernetes operator ProxyGroup/usernode, tsrecorder, or k8s-proxy pods with incomplete RBAC; state Secret recreated or garbage-collected during operator upgrades; API server throttling under node churn; custom validating webhooks blocking Secret updates.","solutions":["Read the wrapped error: 403 means grant the operator get/update/patch on the state Secret; 404 means the Secret is gone — recreate it or restart the pod so the operator rematerializes it; 429/503/timeout — retry","Verify the pod's service account Role/RoleBinding covers secrets get, update, patch for the state Secret in its namespace","Confirm the Secret the operator created for this node (e.g. tailscale-state-*) still exists and matches what the pod was started with","Retry SetInitialKeys with backoff; transient apiserver errors resolve on their own"],"exampleFix":"// before\nif err := state.SetInitialKeys(store, podUID); err != nil {\n\tlog.Fatalf(\"fatal: %v\", err)\n}\n\n// after\nvar se *k8sapierrors.StatusError\nif errors.As(err, &se) {\n\tswitch se.Status().Code {\n\tcase 403:\n\t\t// RBAC: surface actionable message, operator Role needs secrets update\n\tcase 404:\n\t\t// state Secret deleted: recreate or restart the pod\n\t}\n}\n// otherwise: transient apiserver failure, retry with backoff","handlingStrategy":"retry","validationCode":"// Before starting, verify the state Secret is at least readable;\n// write failures then narrow to permissions/apiserver health.\nif _, err := clientset.CoreV1().Secrets(ns).Get(ctx, secretName, metav1.GetOptions{}); err != nil {\n\treturn fmt.Errorf(\"state secret %s/%s unavailable: %w\", ns, secretName, err)\n}","typeGuard":null,"tryCatchPattern":"if err := state.SetInitialKeys(store, podUID); err != nil {\n\tvar se *k8sapierrors.StatusError\n\tif errors.As(err, &se) {\n\t\tswitch se.Status().Code {\n\t\tcase 403:\n\t\t\t// permanent: RBAC — do not retry, alert\n\t\tcase 404:\n\t\t\t// Secret gone: recreate or restart pod\n\t\tdefault:\n\t\t\t// transient: retry with backoff\n\t\t}\n\t}\n}","preventionTips":["Deploy the operator's full RBAC (Role + RoleBinding) for secrets get/update/patch before running client pods","Keep the pod restartable: SetInitialKeys is idempotent, so retries are safe","Monitor for state Secret deletions in the operator namespace"],"tags":["kubernetes","tailscale","state-store","rbac","secrets","operator"],"backgroundTag":"kubernetes-secret-update-failed","analyzedSha":"6e0912f97994f927632b34ae9e63b53d6516a6ac","analyzedAt":"2026-08-18T08:17:25.280Z","contentChangedAt":"2026-08-18T08:17:25.280Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}