{"record":{"id":"874fa7e345e4609a","repo":"hyperledger/fabric","slug":"orderer-s-hung-up-without-sending-status","errorCode":null,"errorMessage":"orderer `%s` hung up without sending status","messagePattern":"orderer `(.+?)` hung up without sending status","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"common/deliverclient/blocksprovider/block_receiver.go","lineNumber":103,"sourceCode":"\tclose(br.stopC)\n\tbr.logger.Infof(\"BlockReceiver stopped\")\n}\n\n// ProcessIncoming processes incoming messages until stopped or encounters an error.\nfunc (br *BlockReceiver) ProcessIncoming(onSuccess func(blockNum uint64, channelConfig *common.Config)) error {\n\tvar err error\n\nRecvLoop: // Loop until the endpoint is refreshed, or there is an error on the connection\n\tfor {\n\t\tselect {\n\t\tcase <-br.endpoint.Refreshed:\n\t\t\tbr.logger.Infof(\"Ordering endpoints have been refreshed, disconnecting from deliver to reconnect using updated endpoints\")\n\t\t\terr = &errRefreshEndpoint{message: fmt.Sprintf(\"orderer endpoint `%s` has been refreshed, \", br.endpoint.Address)}\n\t\t\tbreak RecvLoop\n\t\tcase response, ok := <-br.recvC:\n\t\t\tif !ok {\n\t\t\t\tbr.logger.Warningf(\"Orderer hung up without sending status\")\n\t\t\t\terr = errors.Errorf(\"orderer `%s` hung up without sending status\", br.endpoint.Address)\n\t\t\t\tbreak RecvLoop\n\t\t\t}\n\t\t\tvar blockNum uint64\n\t\t\tvar channelConfig *common.Config\n\t\t\tblockNum, channelConfig, err = br.processMsg(response)\n\t\t\tif err != nil {\n\t\t\t\tbr.logger.Warningf(\"Got error while attempting to receive blocks: %v\", err)\n\t\t\t\terr = errors.WithMessagef(err, \"got error while attempting to receive blocks from orderer `%s`\", br.endpoint.Address)\n\t\t\t\tbreak RecvLoop\n\t\t\t}\n\t\t\tonSuccess(blockNum, channelConfig)\n\t\tcase <-br.stopC:\n\t\t\tbr.logger.Infof(\"BlockReceiver got a signal to stop\")\n\t\t\terr = &ErrStopping{Message: \"got a signal to stop\"}\n\t\t\tbreak RecvLoop\n\t\t}\n\t}\n","sourceCodeStart":85,"sourceCodeEnd":121,"githubUrl":"https://github.com/hyperledger/fabric/blob/2736b63f8fd5932511d56fe68b7039d15977f7f6/common/deliverclient/blocksprovider/block_receiver.go#L85-L121","documentation":"ProcessIncoming reads deliver responses from br.recvC; when the channel closes (ok == false) the orderer terminated the deliver stream without sending a DeliveredStatus/Status message. The receiver logs a warning and returns errors.Errorf(\"orderer `%s` hung up without sending status\") with the endpoint address. This aborts the current receive loop for that endpoint.","triggerScenarios":"In FetchBlocks/DeliverBlocks, the gRPC deliver stream's Recv channel closes without delivering a final status: orderer process crash/restart, gRPC connection teardown, idle-timeout or max-stream limits, or an abrupt server-side disconnect mid-stream.","commonSituations":"Orderer pod restarted or crashed during block streaming; load balancer/proxy (e.g. ingress, ALB) killing idle gRPC connections; orderer shutting down for maintenance; network interruptions that close the TCP connection cleanly without a deliver status.","solutions":["Retry the connection — the block fetcher typically rotates to the next orderer endpoint; ensure retry/backoff logic is enabled and endpoints are diverse.","Check orderer logs at the matching timestamp for crashes (panic, OOM kill) or graceful shutdown, and fix the underlying cause (e.g. memory limits).","If behind a proxy/LB, increase idle timeouts or disable connection draining that cuts gRPC streams; use direct orderer access where possible.","Verify keepalive settings on the gRPC client so half-open connections are detected and re-established."],"exampleFix":"// before: single orderer endpoint, stream dies -> fetch fails\nordererEndpoints:\n  - orderer1.example.com:7050\n// after: multiple endpoints so the fetcher reconnects to another orderer\nordererEndpoints:\n  - orderer1.example.com:7050\n  - orderer2.example.com:7050\n  - orderer3.example.com:7050","handlingStrategy":"retry","validationCode":"// health-check the orderer stream target before ProcessIncoming\nfunc ordererHealthy(addr string) error {\n    ctx, cancel := context.WithTimeout(context.Background(), 3*time.Second)\n    defer cancel()\n    cc, err := grpc.DialContext(ctx, addr, grpc.WithBlock(), grpc.WithTransportCredentials(creds()))\n    if err != nil {\n        return fmt.Errorf(\"orderer %s unhealthy: %w\", addr, err)\n    }\n    cc.Close()\n    return nil\n}","typeGuard":null,"tryCatchPattern":"err := br.ProcessIncoming(ctx)\nif err != nil {\n    if strings.Contains(err.Error(), \"hung up without sending status\") {\n        log.Warnf(\"orderer stream dropped (%v); rotating endpoint and retrying\", err)\n        return retryWithNextEndpoint(ctx)\n    }\n    return err\n}","preventionTips":["Configure gRPC keepalive to detect and recover dropped streams quickly.","Avoid proxies/LBs with short idle timeouts in front of orderer deliver endpoints, or raise their timeouts.","Ensure stable orderer infrastructure (adequate memory limits, graceful shutdown handling).","Keep a multi-orderer endpoint list so the fetcher can reconnect elsewhere after a hang-up."],"tags":["network","grpc","stream","ordering-service","fabric"],"backgroundTag":"connection-closed-without-response","analyzedSha":"2736b63f8fd5932511d56fe68b7039d15977f7f6","analyzedAt":"2026-09-04T08:52:36.465Z","contentChangedAt":"2026-09-04T08:52:36.465Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}