tailscale/tailscale · error

error writing capability version to state store: %w

Error message

error writing capability version to state store: %w

What it means

Thrown by SetInitialKeys in tailscale.com/kube/state: after the pod-UID key was written, writing the capability-version key ('cap-ver', tailcfg.CurrentCapabilityVersion) via store.WriteState failed. In operator-managed Kubernetes deployments the store is ipn/store/kubestore.Store, so this is a Secret update against the API server. The wrapped error carries the exact API failure (status code, RBAC reason, or transport error).

Source

Thrown at kube/state/state.go:39

	"tailscale.com/tailcfg"
	"tailscale.com/util/deephash"
)

const (
	keyPodUID     = ipn.StateKey(kubetypes.KeyPodUID)
	keyCapVer     = ipn.StateKey(kubetypes.KeyCapVer)
	keyDeviceID   = ipn.StateKey(kubetypes.KeyDeviceID)
	keyDeviceIPs  = ipn.StateKey(kubetypes.KeyDeviceIPs)
	keyDeviceFQDN = ipn.StateKey(kubetypes.KeyDeviceFQDN)
)

// SetInitialKeys sets Pod UID and cap ver.
func SetInitialKeys(store ipn.StateStore, podUID string) error {
	if err := store.WriteState(keyPodUID, []byte(podUID)); err != nil {
		return fmt.Errorf("error writing pod UID to state store: %w", err)
	}
	if err := store.WriteState(keyCapVer, fmt.Appendf(nil, "%d", tailcfg.CurrentCapabilityVersion)); err != nil {
		return fmt.Errorf("error writing capability version to state store: %w", err)
	}

	return nil
}

// KeepKeysUpdated sets state store keys consistent with containerboot to
// signal proxy readiness to the operator. It runs until its context is
// cancelled or it hits an error. It watches the IPN bus for SelfChange
// notifications (which fire whenever the self node changes) and reads
// the new self node directly from the notify.
func KeepKeysUpdated(ctx context.Context, store ipn.StateStore, lc klc.LocalClient) error {
	w, err := lc.WatchIPNBus(ctx, ipn.NotifyInitialNetMap)
	if err != nil {
		return fmt.Errorf("error watching IPN bus: %w", err)
	}
	defer w.Close()

	var currentDeviceID, currentDeviceIPs, currentDeviceFQDN deephash.Sum

View on GitHub (pinned to 6e0912f979)

Solutions

1. Read the wrapped error: 403 means grant the operator get/update/patch on the state Secret; 404 means the Secret is gone — recreate it or restart the pod so the operator rematerializes it; 429/503/timeout — retry
2. Verify the pod's service account Role/RoleBinding covers secrets get, update, patch for the state Secret in its namespace
3. Confirm the Secret the operator created for this node (e.g. tailscale-state-*) still exists and matches what the pod was started with
4. Retry SetInitialKeys with backoff; transient apiserver errors resolve on their own

Example fix

// before
if err := state.SetInitialKeys(store, podUID); err != nil {
	log.Fatalf("fatal: %v", err)
}

// after
var se *k8sapierrors.StatusError
if errors.As(err, &se) {
	switch se.Status().Code {
	case 403:
		// RBAC: surface actionable message, operator Role needs secrets update
	case 404:
		// state Secret deleted: recreate or restart the pod
	}
}
// otherwise: transient apiserver failure, retry with backoff

// Before starting, verify the state Secret is at least readable;
// write failures then narrow to permissions/apiserver health.
if _, err := clientset.CoreV1().Secrets(ns).Get(ctx, secretName, metav1.GetOptions{}); err != nil {
	return fmt.Errorf("state secret %s/%s unavailable: %w", ns, secretName, err)
}

if err := state.SetInitialKeys(store, podUID); err != nil {
	var se *k8sapierrors.StatusError
	if errors.As(err, &se) {
		switch se.Status().Code {
		case 403:
			// permanent: RBAC — do not retry, alert
		case 404:
			// Secret gone: recreate or restart pod
		default:
			// transient: retry with backoff
		}
	}
}

When it happens

Trigger: Calling state.SetInitialKeys(store, podUID) when the kubestore Secret update fails: RBAC lacking update/patch on secrets, the node's state Secret deleted mid-run, API server unreachable/429/timeout, or an admission webhook rejecting the write. The preceding pod-UID write succeeded, so the failure raced in between the two writes.

Common situations: Tailscale Kubernetes operator ProxyGroup/usernode, tsrecorder, or k8s-proxy pods with incomplete RBAC; state Secret recreated or garbage-collected during operator upgrades; API server throttling under node churn; custom validating webhooks blocking Secret updates.

Related errors

• failed to store device ID in state: %w
• failed to store device IPs in state: %w
• failed to store device FQDN in state: %w
• creating state directory: %w
• failed to open %q: %w

AI-assisted analysis of tailscale/tailscale@6e0912f979 (2026-08-18). Data as JSON: /api/errors/da8db3c94cf3bda1. Report an issue: GitHub.