hyperledger/fabric · error

timed out (%v) waiting on forwarding to %d

Error message

timed out (%v) waiting on forwarding to %d

What it means

Returned when forwarding a Submit request to the Raft leader does not get acknowledged within c.opts.RPCTimeout. The forwarded RPC was sent but no confirmation arrived before the timer fired, typically because the leader is slow, overloaded, or unreachable.

Source

Thrown at orderer/consensus/etcdraft/chain.go:597

	sentChan := make(chan struct{})
	atomicErr := &atomic.Value{}

	report := func(err error) {
		if err != nil {
			atomicErr.Store(err.Error())
			c.Metrics.ProposalFailures.Add(1)
		}
		close(sentChan)
	}

	c.rpc.SendSubmit(lead, req, report)

	select {
	case <-sentChan:
	case <-c.doneC:
		return errors.Errorf("chain is stopped")
	case <-timer.C:
		return errors.Errorf("timed out (%v) waiting on forwarding to %d", c.opts.RPCTimeout, lead)
	}

	if atomicErr.Load() != nil {
		return errors.New(atomicErr.Load().(string))
	}
	return nil
}

type apply struct {
	entries []*raftpb.Entry
	soft    *raft.SoftState
}

func isCandidate(state raft.StateType) bool {
	return state == raft.StatePreCandidate || state == raft.StateCandidate
}

func (c *Chain) run() {

View on GitHub (pinned to 2736b63f8f)

Solutions

1. Retry the submission (client SDK retry) — the node may re-forward to the new leader.
2. Increase RPCTimeout in the orderer General.Keepalive/raft configuration if timeouts occur under normal latency.
3. Check network connectivity and TLS between orderer nodes; measure RTT to the leader.
4. Inspect leader CPU/memory/disk pressure and scale the ordering service if overloaded.

Example fix

// orderer.yaml
General:
  # before
  RPCTimeout: 3s
  # after
  RPCTimeout: 10s

if err.Error() == "chain is stopped" {
    return retrySubmit(req, nextOrderer)
}

When it happens

Trigger: rpc.SendSubmit to the leader does not complete before RPCTimeout elapses — network latency/partition, leader overloaded, TLS handshake delays, or leader change mid-forward.

Common situations: High transaction load saturating the leader; network congestion between datacenters hosting orderers; leader being evicted/replaced while forwarding; overly small GeneralConfig.RPCTimeout setting.

Understand the failure class

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• cannot load client cert for consenter %s:%d: %s
• cannot load server cert for consenter %s:%d: %s
• timeout expired while executing transaction
• failed to unmarshal ClusterMetadata: %s
• no Raft leader

AI-assisted analysis of hyperledger/fabric@2736b63f8f (2026-09-04). Data as JSON: /api/errors/cb5a1fb3e0563235. Report an issue: GitHub.