{"record":{"id":"c7e16c648c73833a","repo":"hyperledger/fabric","slug":"stream-d-is-stale","errorCode":null,"errorMessage":"stream %d is stale","messagePattern":"stream (.+?) is stale","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"orderer/common/cluster/clusterservice.go","lineNumber":201,"sourceCode":"func (s *ClusterService) handleMessage(stream ClusterStepStream, addr string, exp *certificateExpirationCheck, channel string, sender uint64, streamID uint64) error {\n\trequest, err := stream.Recv()\n\tif err == io.EOF {\n\t\treturn err\n\t}\n\tif err != nil {\n\t\ts.Logger.Warningf(\"Stream read from %s failed: %v\", addr, err)\n\t\treturn err\n\t}\n\tif request == nil {\n\t\treturn errors.Errorf(\"request message is nil\")\n\t}\n\n\ts.Lock.RLock()\n\t_, authorized := s.MembershipByChannel[channel].AuthorizedStreams.Load(streamID)\n\ts.Lock.RUnlock()\n\n\tif !authorized {\n\t\treturn errors.Errorf(\"stream %d is stale\", streamID)\n\t}\n\n\tif s.StepLogger.IsEnabledFor(zap.DebugLevel) {\n\t\tnodeName := commonNameFromContext(stream.Context())\n\t\ts.StepLogger.Debugf(\"Received message from %s(%s): %v\", nodeName, addr, clusterRequestAsString(request))\n\t}\n\n\texp.checkExpiration(time.Now(), channel)\n\n\tif tranReq := request.GetNodeTranrequest(); tranReq != nil {\n\t\tsubmitReq := &orderer.SubmitRequest{\n\t\t\tChannel:           channel,\n\t\t\tLastValidationSeq: tranReq.LastValidationSeq,\n\t\t\tPayload:           tranReq.Payload,\n\t\t}\n\t\treturn s.RequestHandler.OnSubmit(channel, sender, submitReq)\n\t} else if clusterConReq := request.GetNodeConrequest(); clusterConReq != nil {\n\t\tconReq := &orderer.ConsensusRequest{","sourceCodeStart":183,"sourceCodeEnd":219,"githubUrl":"https://github.com/hyperledger/fabric/blob/2736b63f8fd5932511d56fe68b7039d15977f7f6/orderer/common/cluster/clusterservice.go#L183-L219","documentation":"ClusterService tracks, per channel, which streams were authorized via DispatchConsensus/DispatchSubmit; each successfully authorized request registers its streamID in MembershipByChannel[channel].AuthorizedStreams. When a subsequent message arrives on a stream that was never authorized — or whose authorization was evicted — handleMessage rejects it with 'stream N is stale', since allowing it could let an unauthenticated node inject consensus/submit messages.","triggerScenarios":"A Step stream is opened and the first message on it arrives after the authorization window closed (e.g. membership recomputed and the sender removed), or a request is sent on a stream that never had its first Submit/Consensus authorized — typically due to auth failures on the first message, rapid reconnects, or channel membership changes removing the remote node.","commonSituations":"Orderer removed from the channel's consenters set while it still holds an open stream; TLS cert rotation in progress; transient network partitions causing the sender to keep pushing on a connection whose authorization the receiver already dropped; clock/leader changes during leader election (etcdraft step-down/step-up churn).","solutions":["Confirm the sending orderer is still in the channel's consenter set; if it was removed intentionally, restart it or remove it from the network.","Reconnect: the sending side detects the stale-stream error and dials a fresh stream, re-arming authorization — ensure retries are enabled (default Cluster.DialTimeout/Retry).","Fix the underlying auth failure on the stream's first message (cert mismatch, signature failure) so the stream registers as authorized.","Verify channel membership consistency: all orderers should agree on the channel config; run a config update if one node is out of sync.","Check for rapid reconnect loops (misbehaving dialer, aggressive idle timeouts) and tune keepalive/timeout settings."],"exampleFix":"// before: sender keeps writing on a dropped-authorization stream\n// after: let the client re-dial on stale-stream errors\nerr := stream.Send(req)\nif err != nil /* or peer returns stale */ {\n    conn.Close()\n    conn = dialer.Dial(channel, endpoint) // fresh stream gets re-authorized\n    stream = conn.NewStream()\n}","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"err := handleMessage(stream, addr, exp, channel, sender, streamID)\nif err != nil && strings.Contains(err.Error(), \"is stale\") {\n    // stream lost authorization: close and re-dial; the fresh stream re-arms authorization\n    conn.Close()\n    return reconnectAndStep(channel, endpoint)\n}","preventionTips":["Keep every orderer in the channel consenter set in sync with the network reality","Complete cert rotations on all nodes simultaneously to avoid authorization windows closing mid-stream","Tune Cluster.DialTimeout/RetryTime so senders recover promptly after a stale-stream error","Alert on frequent stale-stream errors — they usually signal membership drift or an underlying auth failure"],"tags":["hyperledger-fabric","orderer","raft","stream","authorization"],"backgroundTag":"stale-stream-authorization","analyzedSha":"2736b63f8fd5932511d56fe68b7039d15977f7f6","analyzedAt":"2026-09-04T08:52:36.465Z","contentChangedAt":"2026-09-04T08:52:36.465Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}