hyperledger/fabric · critical

failed to read Raft metadata

Error message

failed to read Raft metadata

What it means

HandleChain (orderer/consensus/etcdraft/consenter.go:142) calls ReadBlockMetadata to reconcile the raft metadata stored in the last block's metadata field with the consenters declared in the channel config. This error wraps a mismatch: counts differ, a config consenter is missing from block metadata, or an ID is out of range. It prevents starting with an inconsistent consenter-to-ID mapping after restart or config change.

Source

Thrown at orderer/consensus/etcdraft/consenter.go:142

	if m.GetOptions() == nil {
		return nil, errors.New("etcdraft options have not been provided")
	}

	isMigration := (metadata == nil || len(metadata.GetValue()) == 0) && (support.Height() > 1)
	if isMigration {
		c.Logger.Debugf("Block metadata is nil at block height=%d, it is consensus-type migration", support.Height())
	}

	// determine raft replica set mapping for each node to its id
	// for newly started chain we need to read and initialize raft
	// metadata by creating mapping between conseter and its id.
	// In case chain has been restarted we restore raft metadata
	// information from the recently committed block meta data
	// field.
	blockMetadata, err := ReadBlockMetadata(metadata, m)
	if err != nil {
		return nil, errors.Wrapf(err, "failed to read Raft metadata")
	}

	consenters := CreateConsentersMap(blockMetadata, m)

	id, err := c.detectSelfID(consenters)
	if err != nil {
		return nil, errors.Wrap(err, "without a system channel, a follower should have been created")
	}

	var evictionSuspicion time.Duration
	if c.EtcdRaftConfig.EvictionSuspicion == "" {
		c.Logger.Infof("EvictionSuspicion not set, defaulting to %v", DefaultEvictionSuspicion)
		evictionSuspicion = DefaultEvictionSuspicion
	} else {
		evictionSuspicion, err = time.ParseDuration(c.EtcdRaftConfig.EvictionSuspicion)
		if err != nil {
			c.Logger.Panicf("Failed parsing Consensus.EvictionSuspicion: %s: %v", c.EtcdRaftConfig.EvictionSuspicion, err)
		}

View on GitHub (pinned to 2736b63f8f)

Solutions

1. Pull and commit the latest blocks from the ordering service so the node sees the config update block before starting raft
2. Restore the orderer's ledger/filesystem from a consistent backup taken after the last config change
3. Compare the number of block-metadata consenters with ConfigMetadata consenters; both must match one-to-one
4. For irreparable local state, rejoin the node with a fresh genesis and sync (or re-add it via config update)

Example fix

// before: starting node with stale ledger missing new consenter
# ledger height < config block adding consenter -> counts mismatch
// after
# fetch latest blocks: peer/orderer resync from service endpoint, then restart orderer

blockMetadata, err := ReadBlockMetadata(metadata, configMetadata)
if err != nil {
    // resync ledger from the ordering service or restore consistent backup before starting raft
    return fmt.Errorf("block/config metadata mismatch, resync needed: %w", err)
}

func metadataConsistent(blockMeta []*common.Metadata, cfg *etcdraft.ConfigMetadata) bool {
    return len(blockMeta) == len(cfg.GetConsenters())
}

chain, err := consenter.HandleChain(support, metadata)
if err != nil {
    if strings.Contains(err.Error(), "failed to read Raft metadata") {
        // fetch latest blocks / restore consistent backup, then retry startup
    }
}

When it happens

Trigger: Orderer restart/rejoin where the last committed block's raft metadata disagrees with the channel config's ConfigMetadata consenters (different number of consenters, unknown consenter, or ID not in [1, len]).

Common situations: Changing the consenter set in config while the local node is behind (has not committed the config block); restoring an orderer's ledger from a backup older than a config update; corrupted or truncated block metadata after crash.

Related errors

• invalid new config metadata
• failed to unmarshal consensus metadata
• nil metadata
• failed to unmarshal updated (new) etcdraft metadata configur
• no such block hash [%x] in index

AI-assisted analysis of hyperledger/fabric@2736b63f8f (2026-09-04). Data as JSON: /api/errors/525559189b47a019. Report an issue: GitHub.