hyperledger/fabric · critical

failed to restore persisted raft data: %s

Error message

failed to restore persisted raft data: %s

What it means

During chain creation, NewChain calls CreateStorage to open the raft WAL and snapshot store. If restoring the persisted raft data fails (corrupt or unreadable WAL/snapshot files), the chain cannot start and the underlying error is wrapped with 'failed to restore persisted raft data'. This guards against starting raft from inconsistent persistence that could fork the channel.

Source

Thrown at orderer/consensus/etcdraft/chain.go:236

}

// NewChain constructs a chain object.
func NewChain(
	support consensus.ConsenterSupport,
	opts Options,
	conf Configurator,
	rpc RPC,
	cryptoProvider bccsp.BCCSP,
	f CreateBlockPuller,
	haltCallback func(),
	observeC chan<- raft.SoftState,
) (*Chain, error) {
	lg := opts.Logger.With("channel", support.ChannelID(), "node", opts.RaftID)

	fresh := !wal.Exist(opts.WALDir)
	storage, err := CreateStorage(lg, opts.WALDir, opts.SnapDir, opts.MemoryStorage)
	if err != nil {
		return nil, errors.Errorf("failed to restore persisted raft data: %s", err)
	}

	if opts.SnapshotCatchUpEntries == 0 {
		storage.SnapshotCatchUpEntries = DefaultSnapshotCatchUpEntries
	} else {
		storage.SnapshotCatchUpEntries = opts.SnapshotCatchUpEntries
	}

	sizeLimit := opts.SnapshotIntervalSize
	if sizeLimit == 0 {
		sizeLimit = DefaultSnapshotIntervalSize
	}

	// get block number in last snapshot, if exists
	var snapBlkNum uint64
	cc := &raftpb.ConfState{}
	if s := storage.Snapshot(); !raft.IsEmptySnap(s) {
		b := protoutil.UnmarshalBlockOrPanic(s.GetData())

View on GitHub (pinned to 2736b63f8f)

Solutions

1. Read the wrapped %s error in the log to identify the exact WAL/snapshot failure (e.g., 'corrupt', 'permission denied').
2. If WAL is corrupted beyond repair, restore the channel from the latest valid snapshot or by removing the channel and re-joining from a healthy orderer (ledger will re-sync).
3. Check filesystem permissions on WALDir and SnapDir for the orderer process user, and confirm disk space.
4. Never hand-copy WAL/snap dirs between orderers; instead re-join the channel via the participation API.

Example fix

// before: blindly reusing a corrupt WAL dir
// orderer fails at startup

// after: re-join channel to rebuild raft persistence
// os -remove channel then join again:
// curl -X POST .../participation/v1/channels -d '{...join...}'

if _, err := CreateStorage(lg, walDir, snapDir, memStore); err != nil {
    logger.Errorf("raft data corrupt: %s; re-join channel to rebuild", err)
    // fallback: remove and re-join the channel via participation API
}

When it happens

Trigger: HandleChain/NewChain invoked at orderer startup for a channel whose WALDir or SnapDir contains corrupted, partially-written, or incompatible etcd/wal files — CreateStorage (wal.Open + snapshot store) returns an error.

Common situations: Disk full during write leaving a truncated WAL segment; manual copying of WAL directories between nodes; running a newer/older etcd/raft library with an incompatible WAL format; permission problems reading WALDir/SnapDir.

Related errors

• unable to retrieve block [%d]
• failed to get last block
• chain is stopped
• chain is not started
• failed to process Raft Step message: %s

AI-assisted analysis of hyperledger/fabric@2736b63f8f (2026-09-04). Data as JSON: /api/errors/20a7fc2a406b1b64. Report an issue: GitHub.