{"record":{"id":"7dd89a746ab8066b","repo":"etcd-io/etcd","slug":"failed-to-recover-store-from-backend","errorCode":null,"errorMessage":"failed to recover store from backend","messagePattern":"failed to recover store from backend","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"server/storage/mvcc/kvstore.go","lineNumber":131,"sourceCode":"\ts.hashes = NewHashStorage(lg, s)\n\ts.ReadView = &readView{s}\n\ts.WriteView = &writeView{s}\n\tif s.le != nil {\n\t\ts.le.SetRangeDeleter(func() lease.TxnDelete { return s.Write(traceutil.TODO()) })\n\t}\n\n\ttx := s.b.BatchTx()\n\ttx.LockOutsideApply()\n\ttx.UnsafeCreateBucket(schema.Key)\n\tschema.UnsafeCreateMetaBucket(tx)\n\ttx.Unlock()\n\ts.b.ForceCommit()\n\n\ts.mu.Lock()\n\tdefer s.mu.Unlock()\n\tif err := s.restore(); err != nil {\n\t\t// TODO: return the error instead of panic here?\n\t\tpanic(\"failed to recover store from backend\")\n\t}\n\n\treturn s\n}\n\nfunc (s *store) compactBarrier(ctx context.Context, ch chan struct{}) {\n\tif ctx == nil || ctx.Err() != nil {\n\t\tselect {\n\t\tcase <-s.stopc:\n\t\tdefault:\n\t\t\t// fix deadlock in mvcc, for more information, please refer to pr 11817.\n\t\t\t// s.stopc is only updated in restore operation, which is called by apply\n\t\t\t// snapshot call, compaction and apply snapshot requests are serialized by\n\t\t\t// raft, and do not happen at the same time.\n\t\t\ts.mu.Lock()\n\t\t\tf := schedule.NewJob(\"kvstore_compactBarrier\", func(ctx context.Context) { s.compactBarrier(ctx, ch) })\n\t\t\ts.fifoSched.Schedule(f)\n\t\t\ts.mu.Unlock()","sourceCodeStart":113,"sourceCodeEnd":149,"githubUrl":"https://github.com/etcd-io/etcd/blob/f744d457f484e9f748a0700b48ef96dcf792df33/server/storage/mvcc/kvstore.go#L113-L149","documentation":"This panic fires in mvcc store construction (newStore) when s.restore() fails while rebuilding the mvcc state (current revision, compact revision, keyspace metadata) from the bbolt backend. The TODO in the source ('return the error instead of panic here?') shows the authors consider it a fatal, unrecoverable store-state problem: the backend file exists but its contents cannot be interpreted, so the store cannot start.","triggerScenarios":"Opening a backend whose 'key'/'meta' buckets contain inconsistent or unreadable state: s.restore() errors on a corrupted bbolt file, a data directory written by an incompatible etcd version, a partially written backend after a crash (torn write during ForceCommit), or a data dir manually copied/truncated while the process was running.","commonSituations":"Disk full or host crash during heavy write load leaving a torn bbolt file; restoring a data-dir snapshot taken with cp instead of etcdutl snapshot restore; downgrading a member to an older etcd whose mvcc schema differs; mixing data dirs between cluster members; bbolt file truncated by backup tooling.","solutions":["Check backend integrity first: run `etcdutl bbolt check ./member/snap/db` (or go run go.etcd.io/bbolt/cmd/bbolt info) against the data dir's db file to confirm physical corruption.","If corruption is confirmed, restore the member from a snapshot: `etcdutl snapshot restore snapshot.db --data-dir ...` and restart the member.","If the db is intact but unreadable, verify version compatibility — a data dir written by a newer etcd minor cannot be opened by an older binary; run the matching or newer etcd version.","As a last resort on a multi-member cluster, remove the member's data dir and re-add it (etcdctl member remove/add) letting it catch up from peers."],"exampleFix":null,"handlingStrategy":"fallback","validationCode":"# Before starting etcd on an existing data dir, verify backend integrity:\netcdutl bbolt check /var/lib/etcd/member/snap/db","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Take snapshots with etcdctl snapshot save and restore with etcdutl snapshot restore — never cp a live data dir.","Run etcd under a supervisor (systemd) that restarts it after a crash so the WAL/backend recovery path completes.","Monitor disk space; bbolt torn writes often follow ENOSPC.","Never downgrade a member below the version that wrote its data dir; check version compatibility before swapping binaries."],"tags":["mvcc","storage","corruption","panic","startup"],"backgroundTag":null,"analyzedSha":"f744d457f484e9f748a0700b48ef96dcf792df33","analyzedAt":"2026-08-15T09:39:50.079Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}