k3s-io/k3s · critical

failed to read consistent index

Error message

failed to read consistent index

What it means

When initializing the v3 storage layer the bbolt backend is opened first (which can itself panic on deep corruption), then schema.ReadConsistentIndex is read in a best-effort way. A zero consistent index means the db exists but contains no usable MVCC state - it is empty, truncated, or inconsistent - so the store refuses to start.

Source

Thrown at pkg/etcd/store/store.go:277

	// open backend database
	bcfg := backend.DefaultBackendConfig(logger)
	bcfg.Path = path
	bcfg.UnsafeNoFsync = true
	bcfg.BatchInterval = time.Hour
	bcfg.BatchLimit = 100000

	// try to open the bbolt database; this may unrecoverably panic from inside
	// the bbolt freelist goroutine if the database is in an inconsistent state.
	s.be = backend.New(bcfg)
	if s.be == nil {
		return nil, errors.New("failed to open database")
	}

	// try to get current index from backend; this may fail if the bbolt database
	// was opened successfully but is in an inconsistent state.
	if currentIndex, _ := schema.ReadConsistentIndex(s.be.ReadTx()); currentIndex == 0 {
		return nil, errors.New("failed to read consistent index")
	}

	// We do not bother checking the latest snapshot index from the WAL or attempting to
	// restore from a snapshot, as v3 store snapshots are only created when replicas are
	// lagging and the leader sends them a fresh copy of the bbolt database - and are
	// therefore highly unlikely to exist. The .snap files in the snap dir are for the
	// legacy v2 store, and are of no use.
	//
	// ref: https://etcd.io/docs/v3.6/learning/persistent-storage-files/#long-leaving-files
	// > Note: Periodic snapshots generated on each replica are only emitted in the form of
	// > *.snap file (not snap.db file). So there is no guarantee the most recent snapshot (in
	// > WAL log) has the *.snap.db file. But in such a case the backend (snap/db) is expected
	// > to be newer than the snapshot.

	s.kv = mvcc.NewStore(logger, s.be, &lease.FakeLessor{}, mvcc.StoreConfig{})
	logrus.Info("Opened etcd MVCC KV store")

	// nb: closing the kv store does not implicitly close its backend; the backend must be closed separately

View on GitHub (pinned to 6ba341e396)

Solutions

1. Restore from a known-good snapshot: k3s server --cluster-reset --cluster-reset-restore-path=<snapshot>.
2. Or wipe the member's db directory so the member rejoins and receives a fresh copy from the leader (only safe for a non-quorum-critical member).
3. Check filesystem health (dmesg for I/O errors, fsck) and free disk before restarting, or the corruption will recur.

Example fix

# before: restart against a corrupted db (fails with 'failed to read consistent index')
systemctl restart k3s
# after: reset this member from a snapshot (or remove its db to re-replicate)
k3s server --cluster-reset --cluster-reset-restore-path=/var/lib/rancher/k3s/server/db/snapshots/<snap>.zip

st, err := store.Open(...)
if err != nil {
    if strings.Contains(err.Error(), "failed to read consistent index") {
        // db unusable: fall back to snapshot restore or member rejoin
        return restoreFromSnapshotOrRejoin()
    }
    return err
}

When it happens

Trigger: Opening a backend db whose consistent_index key is absent or zero: a zero-length or freshly created file where a populated db was expected; a db truncated by disk-full writes; a db where the WAL/bbolt state was lost but the file itself survived.

Common situations: Nodes that ran out of disk during writes; filesystem corruption after power loss; someone emptying or partially deleting the db dir; copy tools that created placeholder files.

Related errors

• etcd member has status errors: %s
• no bootstrap data found
• %s alarm must be disarmed manually
• no snapshots given for removal
• invalid output format:

AI-assisted analysis of k3s-io/k3s@6ba341e396 (2026-08-15). Data as JSON: /api/errors/4fcb68cbbc1bb036. Report an issue: GitHub.