{"record":{"id":"cb5a1fb3e0563235","repo":"hyperledger/fabric","slug":"timed-out-v-waiting-on-forwarding-to-d","errorCode":null,"errorMessage":"timed out (%v) waiting on forwarding to %d","messagePattern":"timed out \\((.+?)\\) waiting on forwarding to (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"orderer/consensus/etcdraft/chain.go","lineNumber":597,"sourceCode":"\tsentChan := make(chan struct{})\n\tatomicErr := &atomic.Value{}\n\n\treport := func(err error) {\n\t\tif err != nil {\n\t\t\tatomicErr.Store(err.Error())\n\t\t\tc.Metrics.ProposalFailures.Add(1)\n\t\t}\n\t\tclose(sentChan)\n\t}\n\n\tc.rpc.SendSubmit(lead, req, report)\n\n\tselect {\n\tcase <-sentChan:\n\tcase <-c.doneC:\n\t\treturn errors.Errorf(\"chain is stopped\")\n\tcase <-timer.C:\n\t\treturn errors.Errorf(\"timed out (%v) waiting on forwarding to %d\", c.opts.RPCTimeout, lead)\n\t}\n\n\tif atomicErr.Load() != nil {\n\t\treturn errors.New(atomicErr.Load().(string))\n\t}\n\treturn nil\n}\n\ntype apply struct {\n\tentries []*raftpb.Entry\n\tsoft    *raft.SoftState\n}\n\nfunc isCandidate(state raft.StateType) bool {\n\treturn state == raft.StatePreCandidate || state == raft.StateCandidate\n}\n\nfunc (c *Chain) run() {","sourceCodeStart":579,"sourceCodeEnd":615,"githubUrl":"https://github.com/hyperledger/fabric/blob/2736b63f8fd5932511d56fe68b7039d15977f7f6/orderer/consensus/etcdraft/chain.go#L579-L615","documentation":"Returned when forwarding a Submit request to the Raft leader does not get acknowledged within c.opts.RPCTimeout. The forwarded RPC was sent but no confirmation arrived before the timer fired, typically because the leader is slow, overloaded, or unreachable.","triggerScenarios":"rpc.SendSubmit to the leader does not complete before RPCTimeout elapses — network latency/partition, leader overloaded, TLS handshake delays, or leader change mid-forward.","commonSituations":"High transaction load saturating the leader; network congestion between datacenters hosting orderers; leader being evicted/replaced while forwarding; overly small GeneralConfig.RPCTimeout setting.","solutions":["Retry the submission (client SDK retry) — the node may re-forward to the new leader.","Increase RPCTimeout in the orderer General.Keepalive/raft configuration if timeouts occur under normal latency.","Check network connectivity and TLS between orderer nodes; measure RTT to the leader.","Inspect leader CPU/memory/disk pressure and scale the ordering service if overloaded."],"exampleFix":"// orderer.yaml\nGeneral:\n  # before\n  RPCTimeout: 3s\n  # after\n  RPCTimeout: 10s","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"if err.Error() == \"chain is stopped\" {\n    return retrySubmit(req, nextOrderer)\n}","preventionTips":["Avoid restarting orderers during peak traffic.","Keep TLS connections to the leader healthy.","Set client retry/backoff policy.","Watch for leader instability in metrics."],"tags":["hyperledger-fabric","etcdraft","timeout","forwarding"],"backgroundTag":"rpc-timeout","analyzedSha":"2736b63f8fd5932511d56fe68b7039d15977f7f6","analyzedAt":"2026-09-04T08:52:36.465Z","contentChangedAt":"2026-09-04T08:52:36.465Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}