{"record":{"id":"4fcb68cbbc1bb036","repo":"k3s-io/k3s","slug":"failed-to-read-consistent-index","errorCode":null,"errorMessage":"failed to read consistent index","messagePattern":"failed to read consistent index","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"pkg/etcd/store/store.go","lineNumber":277,"sourceCode":"\n\t// open backend database\n\tbcfg := backend.DefaultBackendConfig(logger)\n\tbcfg.Path = path\n\tbcfg.UnsafeNoFsync = true\n\tbcfg.BatchInterval = time.Hour\n\tbcfg.BatchLimit = 100000\n\n\t// try to open the bbolt database; this may unrecoverably panic from inside\n\t// the bbolt freelist goroutine if the database is in an inconsistent state.\n\ts.be = backend.New(bcfg)\n\tif s.be == nil {\n\t\treturn nil, errors.New(\"failed to open database\")\n\t}\n\n\t// try to get current index from backend; this may fail if the bbolt database\n\t// was opened successfully but is in an inconsistent state.\n\tif currentIndex, _ := schema.ReadConsistentIndex(s.be.ReadTx()); currentIndex == 0 {\n\t\treturn nil, errors.New(\"failed to read consistent index\")\n\t}\n\n\t// We do not bother checking the latest snapshot index from the WAL or attempting to\n\t// restore from a snapshot, as v3 store snapshots are only created when replicas are\n\t// lagging and the leader sends them a fresh copy of the bbolt database - and are\n\t// therefore highly unlikely to exist. The .snap files in the snap dir are for the\n\t// legacy v2 store, and are of no use.\n\t//\n\t// ref: https://etcd.io/docs/v3.6/learning/persistent-storage-files/#long-leaving-files\n\t// > Note: Periodic snapshots generated on each replica are only emitted in the form of\n\t// > *.snap file (not snap.db file). So there is no guarantee the most recent snapshot (in\n\t// > WAL log) has the *.snap.db file. But in such a case the backend (snap/db) is expected\n\t// > to be newer than the snapshot.\n\n\ts.kv = mvcc.NewStore(logger, s.be, &lease.FakeLessor{}, mvcc.StoreConfig{})\n\tlogrus.Info(\"Opened etcd MVCC KV store\")\n\n\t// nb: closing the kv store does not implicitly close its backend; the backend must be closed separately","sourceCodeStart":259,"sourceCodeEnd":295,"githubUrl":"https://github.com/k3s-io/k3s/blob/6ba341e396edc16b8dcae978a7c5e3ac7ee5606e/pkg/etcd/store/store.go#L259-L295","documentation":"When initializing the v3 storage layer the bbolt backend is opened first (which can itself panic on deep corruption), then schema.ReadConsistentIndex is read in a best-effort way. A zero consistent index means the db exists but contains no usable MVCC state - it is empty, truncated, or inconsistent - so the store refuses to start.","triggerScenarios":"Opening a backend db whose consistent_index key is absent or zero: a zero-length or freshly created file where a populated db was expected; a db truncated by disk-full writes; a db where the WAL/bbolt state was lost but the file itself survived.","commonSituations":"Nodes that ran out of disk during writes; filesystem corruption after power loss; someone emptying or partially deleting the db dir; copy tools that created placeholder files.","solutions":["Restore from a known-good snapshot: k3s server --cluster-reset --cluster-reset-restore-path=<snapshot>.","Or wipe the member's db directory so the member rejoins and receives a fresh copy from the leader (only safe for a non-quorum-critical member).","Check filesystem health (dmesg for I/O errors, fsck) and free disk before restarting, or the corruption will recur."],"exampleFix":"# before: restart against a corrupted db (fails with 'failed to read consistent index')\nsystemctl restart k3s\n# after: reset this member from a snapshot (or remove its db to re-replicate)\nk3s server --cluster-reset --cluster-reset-restore-path=/var/lib/rancher/k3s/server/db/snapshots/<snap>.zip","handlingStrategy":"fallback","validationCode":null,"typeGuard":null,"tryCatchPattern":"st, err := store.Open(...)\nif err != nil {\n    if strings.Contains(err.Error(), \"failed to read consistent index\") {\n        // db unusable: fall back to snapshot restore or member rejoin\n        return restoreFromSnapshotOrRejoin()\n    }\n    return err\n}","preventionTips":["Keep disk headroom and etcd quota alerts so bbolt never writes while full.","Take regular snapshots so a 'failed to read consistent index' db is a restore, not a disaster.","Avoid partial copies of the db directory between nodes."],"tags":["etcd","bbolt","storage","corruption"],"backgroundTag":null,"analyzedSha":"6ba341e396edc16b8dcae978a7c5e3ac7ee5606e","analyzedAt":"2026-08-15T16:27:54.286Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}