hyperledger/fabric · warning

stream %d is stale

Error message

stream %d is stale

What it means

ClusterService tracks, per channel, which streams were authorized via DispatchConsensus/DispatchSubmit; each successfully authorized request registers its streamID in MembershipByChannel[channel].AuthorizedStreams. When a subsequent message arrives on a stream that was never authorized — or whose authorization was evicted — handleMessage rejects it with 'stream N is stale', since allowing it could let an unauthenticated node inject consensus/submit messages.

Source

Thrown at orderer/common/cluster/clusterservice.go:201

func (s *ClusterService) handleMessage(stream ClusterStepStream, addr string, exp *certificateExpirationCheck, channel string, sender uint64, streamID uint64) error {
	request, err := stream.Recv()
	if err == io.EOF {
		return err
	}
	if err != nil {
		s.Logger.Warningf("Stream read from %s failed: %v", addr, err)
		return err
	}
	if request == nil {
		return errors.Errorf("request message is nil")
	}

	s.Lock.RLock()
	_, authorized := s.MembershipByChannel[channel].AuthorizedStreams.Load(streamID)
	s.Lock.RUnlock()

	if !authorized {
		return errors.Errorf("stream %d is stale", streamID)
	}

	if s.StepLogger.IsEnabledFor(zap.DebugLevel) {
		nodeName := commonNameFromContext(stream.Context())
		s.StepLogger.Debugf("Received message from %s(%s): %v", nodeName, addr, clusterRequestAsString(request))
	}

	exp.checkExpiration(time.Now(), channel)

	if tranReq := request.GetNodeTranrequest(); tranReq != nil {
		submitReq := &orderer.SubmitRequest{
			Channel:           channel,
			LastValidationSeq: tranReq.LastValidationSeq,
			Payload:           tranReq.Payload,
		}
		return s.RequestHandler.OnSubmit(channel, sender, submitReq)
	} else if clusterConReq := request.GetNodeConrequest(); clusterConReq != nil {
		conReq := &orderer.ConsensusRequest{

View on GitHub (pinned to 2736b63f8f)

Solutions

1. Confirm the sending orderer is still in the channel's consenter set; if it was removed intentionally, restart it or remove it from the network.
2. Reconnect: the sending side detects the stale-stream error and dials a fresh stream, re-arming authorization — ensure retries are enabled (default Cluster.DialTimeout/Retry).
3. Fix the underlying auth failure on the stream's first message (cert mismatch, signature failure) so the stream registers as authorized.
4. Verify channel membership consistency: all orderers should agree on the channel config; run a config update if one node is out of sync.
5. Check for rapid reconnect loops (misbehaving dialer, aggressive idle timeouts) and tune keepalive/timeout settings.

Example fix

// before: sender keeps writing on a dropped-authorization stream
// after: let the client re-dial on stale-stream errors
err := stream.Send(req)
if err != nil /* or peer returns stale */ {
    conn.Close()
    conn = dialer.Dial(channel, endpoint) // fresh stream gets re-authorized
    stream = conn.NewStream()
}

err := handleMessage(stream, addr, exp, channel, sender, streamID)
if err != nil && strings.Contains(err.Error(), "is stale") {
    // stream lost authorization: close and re-dial; the fresh stream re-arms authorization
    conn.Close()
    return reconnectAndStep(channel, endpoint)
}

When it happens

Trigger: A Step stream is opened and the first message on it arrives after the authorization window closed (e.g. membership recomputed and the sender removed), or a request is sent on a stream that never had its first Submit/Consensus authorized — typically due to auth failures on the first message, rapid reconnects, or channel membership changes removing the remote node.

Common situations: Orderer removed from the channel's consenters set while it still holds an open stream; TLS cert rotation in progress; transient network partitions causing the sender to keep pushing on a connection whose authorization the receiver already dropped; clock/leader changes during leader election (etcdraft step-down/step-up churn).

Related errors

• request message is nil
• channel members not initialized
• stream %d aborted
• OrdererOrg config does not allow sub-groups
• Attempted to set the batch size preferred max bytes (%v) gre

AI-assisted analysis of hyperledger/fabric@2736b63f8f (2026-09-04). Data as JSON: /api/errors/c7e16c648c73833a. Report an issue: GitHub.