hyperledger/fabric · error

failed to process Raft Step message: %s

Error message

failed to process Raft Step message: %s

What it means

After successful unmarshal, Step forwards the raft message to the embedded etcd/raft node via Node.Step. If the raft library rejects the message (wrong term, invalid sender state, node shutting down), the error is wrapped as 'failed to process Raft Step message'. This typically means consensus state is out of sync or the node is stopping.

Source

Thrown at orderer/consensus/etcdraft/chain.go:522

// Consensus passes the given ConsensusRequest message to the raft.Node instance
func (c *Chain) Consensus(req *orderer.ConsensusRequest, sender uint64) error {
	if err := c.isRunning(); err != nil {
		return err
	}

	stepMsg := &raftpb.Message{}
	if err := proto.Unmarshal(req.GetPayload(), stepMsg); err != nil {
		return fmt.Errorf("failed to unmarshal StepRequest payload to Raft Message: %s", err)
	}

	if stepMsg.GetTo() != c.raftID {
		c.logger.Warnf("Received msg to %d, my ID is probably wrong due to out of date, cowardly halting", stepMsg.GetTo())
		c.halt()
		return nil
	}

	if err := c.Node.Step(context.TODO(), stepMsg); err != nil {
		return fmt.Errorf("failed to process Raft Step message: %s", err)
	}

	if len(req.GetMetadata()) == 0 || atomic.LoadUint64(&c.lastKnownLeader) != sender { // ignore metadata from non-leader
		return nil
	}

	clusterMetadata := &etcdraft.ClusterMetadata{}
	if err := proto.Unmarshal(req.GetMetadata(), clusterMetadata); err != nil {
		return errors.Errorf("failed to unmarshal ClusterMetadata: %s", err)
	}

	c.Metrics.ActiveNodes.Set(float64(len(clusterMetadata.GetActiveNodes())))
	c.ActiveNodes.Store(clusterMetadata.GetActiveNodes())
	c.logger.Infof("Store ActiveNodes %+v", clusterMetadata.GetActiveNodes())

	return nil
}

View on GitHub (pinned to 2736b63f8f)

Solutions

1. Check the wrapped error in the log — 'stopped' indicates shutdown-time traffic and is benign; term conflicts suggest stale state.
2. If state is stale, re-join the channel (remove + join via participation API) or restore from the latest snapshot to rebuild raft state.
3. Ensure removed nodes are deleted from the channel consenter set so peers stop messaging them.
4. Verify WAL/snapshot directories are intact and not shared/replayed from an older run; restart the orderer after fixing persistence.

if err := c.Node.Step(context.TODO(), stepMsg); err != nil {
    if err == raft.ErrStopped {
        logger.Debugf("node stopped; ignoring step message")
        return nil // benign during shutdown
    }
    logger.Warnf("step rejected, raft state may be stale: %s", err)
    return fmt.Errorf("failed to process Raft Step message: %s", err)
}

When it happens

Trigger: Node.Step(context.TODO(), stepMsg) returns an error — messages from a stale/unknown term, steps for a raft node already stopped/evicted, or internal raft state errors after WAL recovery issues.

Common situations: A node rejoined with stale raft state (restored from old WAL) receiving higher-term messages; a removed consenter still receiving messages; raft node torn down while in-flight messages arrive during shutdown.

Related errors

• faulty node, received: %v
• not in the channel
• unable to retrieve block [%d]
• failed to restore persisted raft data: %s
• failed to get last block

AI-assisted analysis of hyperledger/fabric@2736b63f8f (2026-09-04). Data as JSON: /api/errors/a797479480e3150b. Report an issue: GitHub.