etcd-io/etcd · critical

failed to recover store from backend

Error message

failed to recover store from backend

What it means

This panic fires in mvcc store construction (newStore) when s.restore() fails while rebuilding the mvcc state (current revision, compact revision, keyspace metadata) from the bbolt backend. The TODO in the source ('return the error instead of panic here?') shows the authors consider it a fatal, unrecoverable store-state problem: the backend file exists but its contents cannot be interpreted, so the store cannot start.

Source

Thrown at server/storage/mvcc/kvstore.go:131

	s.hashes = NewHashStorage(lg, s)
	s.ReadView = &readView{s}
	s.WriteView = &writeView{s}
	if s.le != nil {
		s.le.SetRangeDeleter(func() lease.TxnDelete { return s.Write(traceutil.TODO()) })
	}

	tx := s.b.BatchTx()
	tx.LockOutsideApply()
	tx.UnsafeCreateBucket(schema.Key)
	schema.UnsafeCreateMetaBucket(tx)
	tx.Unlock()
	s.b.ForceCommit()

	s.mu.Lock()
	defer s.mu.Unlock()
	if err := s.restore(); err != nil {
		// TODO: return the error instead of panic here?
		panic("failed to recover store from backend")
	}

	return s
}

func (s *store) compactBarrier(ctx context.Context, ch chan struct{}) {
	if ctx == nil || ctx.Err() != nil {
		select {
		case <-s.stopc:
		default:
			// fix deadlock in mvcc, for more information, please refer to pr 11817.
			// s.stopc is only updated in restore operation, which is called by apply
			// snapshot call, compaction and apply snapshot requests are serialized by
			// raft, and do not happen at the same time.
			s.mu.Lock()
			f := schedule.NewJob("kvstore_compactBarrier", func(ctx context.Context) { s.compactBarrier(ctx, ch) })
			s.fifoSched.Schedule(f)
			s.mu.Unlock()

View on GitHub (pinned to f744d457f4)

Solutions

1. Check backend integrity first: run `etcdutl bbolt check ./member/snap/db` (or go run go.etcd.io/bbolt/cmd/bbolt info) against the data dir's db file to confirm physical corruption.
2. If corruption is confirmed, restore the member from a snapshot: `etcdutl snapshot restore snapshot.db --data-dir ...` and restart the member.
3. If the db is intact but unreadable, verify version compatibility — a data dir written by a newer etcd minor cannot be opened by an older binary; run the matching or newer etcd version.
4. As a last resort on a multi-member cluster, remove the member's data dir and re-add it (etcdctl member remove/add) letting it catch up from peers.

# Before starting etcd on an existing data dir, verify backend integrity:
etcdutl bbolt check /var/lib/etcd/member/snap/db

When it happens

Trigger: Opening a backend whose 'key'/'meta' buckets contain inconsistent or unreadable state: s.restore() errors on a corrupted bbolt file, a data directory written by an incompatible etcd version, a partially written backend after a crash (torn write during ForceCommit), or a data dir manually copied/truncated while the process was running.

Common situations: Disk full or host crash during heavy write load leaving a torn bbolt file; restoring a data-dir snapshot taken with cp instead of etcdutl snapshot restore; downgrading a member to an older etcd whose mvcc schema differs; mixing data dirs between cluster members; bbolt file truncated by backup tooling.

Related errors

• unexpected revision in delete
• unexpected revision in put
• Unknown result op
• bad compare value
• bad value %v of type %T

AI-assisted analysis of etcd-io/etcd@f744d457f4 (2026-08-15). Data as JSON: /api/errors/7dd89a746ab8066b. Report an issue: GitHub.