hyperledger/fabric · critical

recovery for DB [%s] not possible. Ledger [%s] is created fr

Error message

recovery for DB [%s] not possible. Ledger [%s] is created from a snapshot. Last block in snapshot = [%d], DB needs block [%d] onward

What it means

syncStateAndHistoryDBWithBlockstore returns this when a ledger was created from a snapshot and a state/history database is missing blocks earlier than the snapshot boundary. The DB needs blocks from nextRequiredBlock onward, but the block store (from the snapshot) only contains blocks up to lastBlockInSnapshot, so recovery by replay is impossible. Fabric refuses to proceed rather than produce an inconsistent database.

Source

Thrown at core/ledger/kvledger/kv_ledger.go:399

	}
	lastBlockInBlockStore := info.Height - 1
	recoverables := []recoverable{l.txmgr}
	if l.historyDB != nil {
		recoverables = append(recoverables, l.historyDB)
	}
	recoverers := []*recoverer{}
	for _, recoverable := range recoverables {
		// nextRequiredBlock is nothing but the nextBlockNum expected by the state DB.
		// In other words, the nextRequiredBlock is nothing but the height of stateDB.
		recoverFlag, nextRequiredBlock, err := recoverable.ShouldRecover(lastBlockInBlockStore)
		if err != nil {
			return err
		}

		if l.bootSnapshotMetadata != nil {
			lastBlockInSnapshot := l.bootSnapshotMetadata.LastBlockNumber
			if nextRequiredBlock <= lastBlockInSnapshot {
				return errors.Errorf(
					"recovery for DB [%s] not possible. Ledger [%s] is created from a snapshot. Last block in snapshot = [%d], DB needs block [%d] onward",
					recoverable.Name(),
					l.ledgerID,
					lastBlockInSnapshot,
					nextRequiredBlock,
				)
			}
		}

		if nextRequiredBlock > lastBlockInBlockStore+1 {
			dbName := recoverable.Name()
			return fmt.Errorf("the %s database [height=%d] is ahead of the block store [height=%d]. "+
				"This is possible when the %s database is not dropped after a ledger reset/rollback. "+
				"The %s database can safely be dropped and will be rebuilt up to block store height upon the next peer start",
				dbName, nextRequiredBlock, lastBlockInBlockStore+1, dbName, dbName)
		}
		if recoverFlag {
			recoverers = append(recoverers, &recoverer{nextRequiredBlock, recoverable})

View on GitHub (pinned to 2736b63f8f)

Solutions

1. Drop ALL ledger databases (state, history, bookkeeping) so they rebuild from the snapshot's last block via snapshot verification data
2. Restore the complete, consistent ledgersData set that matches the snapshot
3. If pre-snapshot data is required, resync from genesis instead of using a snapshot

Example fix

// before: only stateDB deleted -> recovery impossible
// after: remove all DBs under ledgersData of the snapshot ledger
//   rm -rf /var/hyperledger/production/ledgersData/state
//   rm -rf /var/hyperledger/production/ledgersData/history
//   rm -rf /var/hyperledger/production/ledgersData/bookkeeper
// then restart peer (rebuilds from snapshot)

// Before mixing snapshot ledgers with DB dirs, ensure DB heights are consistent:
// if the ledger came from a snapshot, ALL DBs must be empty or at snapshot height
if ledgerCreatedFromSnapshot && (stateDBExists || historyDBExists) {
    // drop state, history, and bookkeeper dirs so they rebuild from the snapshot
}

When it happens

Trigger: Peer started with a snapshot-created ledger (bootSnapshotMetadata set) while the state DB or history DB height requires a block number <= the snapshot's last block number (e.g. the DB was deleted/reset, or an older DB was moved in).

Common situations: Operator deletes only the state or history DB directory under ledgersData of a snapshot-bootstrapped peer; restoring an old state DB backup onto a snapshot-based ledger; misconfigured ledgersData where DBs and blockstore come from different sources.

Related errors

• dir %s not empty
• the supplied snapshots appear to be non-comparable. State db
• error decoding the block number
• error decoding the data hash
• error decoding the previous hash

AI-assisted analysis of hyperledger/fabric@2736b63f8f (2026-09-04). Data as JSON: /api/errors/e2fe8ad6e11e455e. Report an issue: GitHub.