k3s-io/k3s · critical

no bootstrap data found in datastore - check server token va

Error message

no bootstrap data found in datastore - check server token value and verify datastore integrity

What it means

Thrown during bootstrap reconciliation when a server using the managed datastore (embedded etcd) retrieved the '/bootstrap' key list but the key selected by getBootstrapKeyFromStorage is nil or has an empty value. The bootstrap key path is '/bootstrap/' + ShortHash(normalizedToken), so the datastore holds no readable bootstrap payload for the supplied token. Without it the server cannot rebuild its CA certs and control-plane runtime.

Source

Thrown at pkg/cluster/bootstrap.go:326

				storageClient = nil
			}
		}

		if storageClient == nil {
			storageClient, err = store.NewTemporaryStore(filepath.Join(c.config.DataDir, "db", "etcd"))
			if err != nil {
				return errors.WithMessage(err, "failed to create temporary datastore client")
			}
		}

		defer storageClient.Close()

		kv, c.saveBootstrap, err = getBootstrapKeyFromStorage(ctx, storageClient, normalizedToken, token)
		if err != nil {
			return err
		}
		if kv == nil || len(kv.Value) == 0 {
			return errors.New("no bootstrap data found in datastore - check server token value and verify datastore integrity")
		}

		dbRawData, err := decrypt(normalizedToken, kv.Value)
		if err != nil {
			return err
		}

		buf = bytes.NewReader(dbRawData)
	}

	if buf == nil {
		return errors.New("no bootstrap data is available to reconcile against")
	}

	paths, err := bootstrap.ObjToMap(crb)
	if err != nil {
		return err
	}

View on GitHub (pinned to 6ba341e396)

Solutions

  1. Verify the token matches the original cluster token: compare with /var/lib/rancher/k3s/server/token (or the first server's token file) and restart with that exact value, including the 'K10...' prefix format.
  2. If the token is correct, inspect the datastore: etcdctl get /bootstrap/ --prefix --keys-only and confirm exactly one key exists whose suffix equals the 12-char ShortHash of the normalized token; remove empty or stale keys written under an old token.
  3. If the datastore is empty/corrupt and this was the only server, restore from an etcd snapshot (k3s etcd-snapshot restore) or re-initialize the cluster: back up and delete the db directory and restart with --cluster-init.
  4. If keys hashed with legacy formats exist (empty-string hash or old token format), start once with a version/token that lets migrateTokens collapse them onto the normalized token key.

Example fix

# before: restarted with a changed token
K3S_TOKEN=NEWTOKEN::server:NEWSECRET k3s server
# after: restart with the token the cluster was created with
K3S_TOKEN=K10xxxxxxxx::server:ORIGINALSECRET k3s server
Defensive patterns

Strategy: validation

Validate before calling

// Before startup (embedding k3s or in a pre-start script): verify the bootstrap key for this token exists and is non-empty.
func bootstrapExists(ctx context.Context, sc store.ReadCloser, token string) (bool, error) {
	norm, err := util.NormalizeToken(token)
	if err != nil { return false, err }
	kv, err := sc.Get(ctx, storageKey(norm), 0)
	if err != nil { return false, err }
	return kv != nil && len(kv.Value) > 0, nil
}
// shell: etcdctl get /bootstrap/ --prefix  # expect exactly one non-empty key

Try / catch

if err := cluster.Bootstrap(...); err != nil {
	if strings.Contains(err.Error(), "no bootstrap data found in datastore") {
		// halt; runbook: verify token against first server's token file, then inspect/restore datastore
	}
	return err
}

Prevention

When it happens

Trigger: Restarting a server with a --token different from the one used at cluster creation (hash maps to a nonexistent key path); the on-disk etcd dir (DataDir/db/etcd) being empty or corrupt when opened via NewTemporaryStore after the local etcd connection fails; a datastore restored from a backup that predates the bootstrap write; a key written with an empty value by an interrupted first start.

Common situations: K3S_TOKEN rotated or typo'd on restart; data-dir moved, partially wiped, or restored from a stale snapshot; mixing k3s versions where token-hash migration left an empty key; single-server cluster whose db directory was deleted while token file remained.

Related errors


AI-assisted analysis of k3s-io/k3s@6ba341e396 (2026-08-15). Data as JSON: /api/errors/9f37c879880e8094. Report an issue: GitHub.