{"record":{"id":"ee52a6c5795dab7c","repo":"diem/diem","slug":"db-corruption-missing-epoch-ending-ledger-info-fo","errorCode":null,"errorMessage":"DB corruption: missing epoch ending ledger info for epoch {}","messagePattern":"DB corruption: missing epoch ending ledger info for epoch (.+?)","errorType":"validation","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"storage/diemdb/src/lib.rs","lineNumber":391,"sourceCode":"            .next_block_epoch();\n        ensure!(\n            end_epoch <= latest_epoch,\n            \"Unable to provide epoch change ledger info for still open epoch. asked upper bound: {}, last sealed epoch: {}\",\n            end_epoch,\n            latest_epoch - 1,  // okay to -1 because genesis LedgerInfo has .next_block_epoch() == 1\n        );\n\n        let (paging_epoch, more) = if end_epoch - start_epoch > limit as u64 {\n            (start_epoch + limit as u64, true)\n        } else {\n            (end_epoch, false)\n        };\n\n        let lis = self\n            .ledger_store\n            .get_epoch_ending_ledger_info_iter(start_epoch, paging_epoch)?\n            .collect::<Result<Vec<_>>>()?;\n        ensure!(\n            lis.len() == (paging_epoch - start_epoch) as usize,\n            \"DB corruption: missing epoch ending ledger info for epoch {}\",\n            lis.last()\n                .map(|li| li.ledger_info().next_block_epoch())\n                .unwrap_or(start_epoch),\n        );\n        Ok((lis, more))\n    }\n\n    fn get_transaction_with_proof(\n        &self,\n        version: Version,\n        ledger_version: Version,\n        fetch_events: bool,\n    ) -> Result<TransactionWithProof> {\n        let proof = self\n            .ledger_store\n            .get_transaction_info_with_proof(version, ledger_version)?;","sourceCodeStart":373,"sourceCodeEnd":409,"githubUrl":"https://github.com/diem/diem/blob/fc4714a8ea273b6efe8b13dbce72ea60aad9a16c/storage/diemdb/src/lib.rs#L373-L409","documentation":"DiemDB verifies that the epoch-ending ledger info iterator returns one entry per epoch in the requested range. If fewer entries are returned than epochs requested, the persistent ledger store is missing epoch-ending records, indicating on-disk data corruption or an incomplete backup/restore. The library throws this instead of returning a partial or inconsistent epoch list to callers.","triggerScenarios":"Calling get_epoch_ending_ledger_infos_impl (via verify_epochs or get_epoch_ending_ledger_infos) when the ledger_store iterator over [start_epoch, paging_epoch) returns a Vec shorter than paging_epoch - start_epoch, i.e. one or more epochs have no ending ledger info row.","commonSituations":"Restoring a node from a partial or pruned backup that skipped epoch-ending ledger infos; RocksDB data files corrupted or truncated; running state sync against a DB written by an older/incompatible format; manually copying a subset of DB column families.","solutions":["Stop the node and verify DB integrity; the epoch-ending ledger info column family is corrupt or incomplete.","Restore the storage directory from a full, verified snapshot/backup that includes epoch-ending ledger infos.","Re-sync the node from scratch from genesis (or latest trusted waypoint) if no good backup exists.","Check disk health (filesystem errors, RocksDB corruption logs) and run db-tool/backup tooling to validate the backup before re-serving queries."],"exampleFix":null,"handlingStrategy":"validation","validationCode":"// before calling get_epoch_ending_ledger_infos\nlet start = start_epoch;\nlet end = start_epoch + num_epochs;\nlet infos = db.get_epoch_ending_ledger_infos(start, end)?;\nif (infos.len() as u64) != num_epochs {\n    return Err(\"epoch-ending ledger info range incomplete; DB likely corrupted\");\n}","typeGuard":null,"tryCatchPattern":"match db.get_epoch_ending_ledger_infos(start, end) {\n    Ok(infos) if infos.len() == expected => Ok(infos),\n    Ok(_) | Err(e @ Error::UnexpectedError(..)) => restore_from_backup_or_resync(),\n}","preventionTips":["Always restore nodes from verified full backups (db-backup tooling), never partial column-family copies.","Monitor disk health and RocksDB corruption logs.","Run DB integrity verification after any restore or storage upgrade before serving traffic.","Keep the node's storage format/migration fully completed before query workloads."],"tags":["storage","database-corruption","epoch"],"backgroundTag":"db-data-corruption","analyzedSha":"fc4714a8ea273b6efe8b13dbce72ea60aad9a16c","analyzedAt":"2026-09-04T21:07:05.890Z","contentChangedAt":"2026-09-04T21:07:05.890Z","schemaVersion":2},"datasetVersion":"2026-09-12T02:17:10.037Z"}