hyperledger/fabric · error

orderer `%s` hung up without sending status

Error message

orderer `%s` hung up without sending status

What it means

ProcessIncoming reads deliver responses from br.recvC; when the channel closes (ok == false) the orderer terminated the deliver stream without sending a DeliveredStatus/Status message. The receiver logs a warning and returns errors.Errorf("orderer `%s` hung up without sending status") with the endpoint address. This aborts the current receive loop for that endpoint.

Source

Thrown at common/deliverclient/blocksprovider/block_receiver.go:103

	close(br.stopC)
	br.logger.Infof("BlockReceiver stopped")
}

// ProcessIncoming processes incoming messages until stopped or encounters an error.
func (br *BlockReceiver) ProcessIncoming(onSuccess func(blockNum uint64, channelConfig *common.Config)) error {
	var err error

RecvLoop: // Loop until the endpoint is refreshed, or there is an error on the connection
	for {
		select {
		case <-br.endpoint.Refreshed:
			br.logger.Infof("Ordering endpoints have been refreshed, disconnecting from deliver to reconnect using updated endpoints")
			err = &errRefreshEndpoint{message: fmt.Sprintf("orderer endpoint `%s` has been refreshed, ", br.endpoint.Address)}
			break RecvLoop
		case response, ok := <-br.recvC:
			if !ok {
				br.logger.Warningf("Orderer hung up without sending status")
				err = errors.Errorf("orderer `%s` hung up without sending status", br.endpoint.Address)
				break RecvLoop
			}
			var blockNum uint64
			var channelConfig *common.Config
			blockNum, channelConfig, err = br.processMsg(response)
			if err != nil {
				br.logger.Warningf("Got error while attempting to receive blocks: %v", err)
				err = errors.WithMessagef(err, "got error while attempting to receive blocks from orderer `%s`", br.endpoint.Address)
				break RecvLoop
			}
			onSuccess(blockNum, channelConfig)
		case <-br.stopC:
			br.logger.Infof("BlockReceiver got a signal to stop")
			err = &ErrStopping{Message: "got a signal to stop"}
			break RecvLoop
		}
	}

View on GitHub (pinned to 2736b63f8f)

Solutions

  1. Retry the connection — the block fetcher typically rotates to the next orderer endpoint; ensure retry/backoff logic is enabled and endpoints are diverse.
  2. Check orderer logs at the matching timestamp for crashes (panic, OOM kill) or graceful shutdown, and fix the underlying cause (e.g. memory limits).
  3. If behind a proxy/LB, increase idle timeouts or disable connection draining that cuts gRPC streams; use direct orderer access where possible.
  4. Verify keepalive settings on the gRPC client so half-open connections are detected and re-established.

Example fix

// before: single orderer endpoint, stream dies -> fetch fails
ordererEndpoints:
  - orderer1.example.com:7050
// after: multiple endpoints so the fetcher reconnects to another orderer
ordererEndpoints:
  - orderer1.example.com:7050
  - orderer2.example.com:7050
  - orderer3.example.com:7050
Defensive patterns

Strategy: retry

Validate before calling

// health-check the orderer stream target before ProcessIncoming
func ordererHealthy(addr string) error {
    ctx, cancel := context.WithTimeout(context.Background(), 3*time.Second)
    defer cancel()
    cc, err := grpc.DialContext(ctx, addr, grpc.WithBlock(), grpc.WithTransportCredentials(creds()))
    if err != nil {
        return fmt.Errorf("orderer %s unhealthy: %w", addr, err)
    }
    cc.Close()
    return nil
}

Try / catch

err := br.ProcessIncoming(ctx)
if err != nil {
    if strings.Contains(err.Error(), "hung up without sending status") {
        log.Warnf("orderer stream dropped (%v); rotating endpoint and retrying", err)
        return retryWithNextEndpoint(ctx)
    }
    return err
}

Prevention

When it happens

Trigger: In FetchBlocks/DeliverBlocks, the gRPC deliver stream's Recv channel closes without delivering a final status: orderer process crash/restart, gRPC connection teardown, idle-timeout or max-stream limits, or an abrupt server-side disconnect mid-stream.

Common situations: Orderer pod restarted or crashed during block streaming; load balancer/proxy (e.g. ingress, ALB) killing idle gRPC connections; orderer shutting down for maintenance; network interruptions that close the TCP connection cleanly without a deliver status.

Related errors

AI-assisted analysis of hyperledger/fabric@2736b63f8f (2026-09-04). Data as JSON: /api/errors/874fa7e345e4609a. Report an issue: GitHub.