{"record":{"id":"92e7a8d9063c5270","repo":"grpc/grpc-go","slug":"connection-active-but-received-health-check-rpc-er","errorCode":null,"errorMessage":"connection active but received health check RPC error: %v","messagePattern":"connection active but received health check RPC error: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"health/client.go","lineNumber":104,"sourceCode":"\t\tif err = s.SendMsg(&healthpb.HealthCheckRequest{Service: service}); err != nil && err != io.EOF {\n\t\t\t// Stream should have been closed, so we can safely continue to create a new stream.\n\t\t\tcontinue retryConnection\n\t\t}\n\t\ts.CloseSend()\n\n\t\tresp := new(healthpb.HealthCheckResponse)\n\t\tfor {\n\t\t\terr = s.RecvMsg(resp)\n\n\t\t\t// Reports healthy for the LBing purposes if health check is not implemented in the server.\n\t\t\tif status.Code(err) == codes.Unimplemented {\n\t\t\t\tsetConnectivityState(connectivity.Ready, nil)\n\t\t\t\treturn err\n\t\t\t}\n\n\t\t\t// Reports unhealthy if server's Watch method gives an error other than UNIMPLEMENTED.\n\t\t\tif err != nil {\n\t\t\t\tsetConnectivityState(connectivity.TransientFailure, fmt.Errorf(\"connection active but received health check RPC error: %v\", err))\n\t\t\t\tcontinue retryConnection\n\t\t\t}\n\n\t\t\t// As a message has been received, removes the need for backoff for the next retry by resetting the try count.\n\t\t\ttryCnt = 0\n\t\t\tif resp.Status == healthpb.HealthCheckResponse_SERVING {\n\t\t\t\tsetConnectivityState(connectivity.Ready, nil)\n\t\t\t} else {\n\t\t\t\tsetConnectivityState(connectivity.TransientFailure, fmt.Errorf(\"connection active but health check failed. status=%s\", resp.Status))\n\t\t\t}\n\t\t}\n\t}\n}\n","sourceCodeStart":86,"sourceCodeEnd":118,"githubUrl":"https://github.com/grpc/grpc-go/blob/03255a9237b6eb32710f6bc4f2de9a675b99fe36/health/client.go#L86-L118","documentation":"Produced by clientHealthCheck when the health-check Watch stream's RecvMsg returns a non-nil error whose code is not Unimplemented. The connection is marked TRANSIENT_FAILURE and the client backs off and retries. This is an expected, recoverable condition reported through the connectivity state machine rather than a fatal error.","triggerScenarios":"Server returns UNAVAILABLE/INTERNAL on the Health/Watch RPC; the underlying transport breaks mid-stream (connection reset, TLS handshake failure, GOAWAY); server implements Health but errors the stream with a non-Unimplemented code.","commonSituations":"Server process restarting or crashing mid-RPC; network blip between client and server; misconfigured TLS causing intermittent failures; server that partially implements health checking; interop with a proxy that terminates long-lived streams.","solutions":["Treat TRANSIENT_FAILURE as transient: the gRPC client already retries with exponential backoff, so no immediate action is needed.","Check server-side logs for the failing Health/Watch RPC to find the root cause.","Verify network stability and TLS configuration between client and server.","If persistent, confirm the server actually serves the gRPC Health service and the requested service name exists."],"exampleFix":"// before: panicking on TRANSIENT_FAILURE\nconn.WaitForStateChange(ctx, connectivity.TransientFailure)\npanic(\"health failed\")\n// after: treat as transient, wait for READY\nfor conn.GetState() != connectivity.Ready {\n    if !conn.WaitForStateChange(ctx, conn.GetState()) {\n        return ctx.Err()\n    }\n}","handlingStrategy":"retry","validationCode":null,"typeGuard":"// Distinguish health-check stream errors from other gRPC errors.\nfunc isHealthCheckTransient(err error) bool {\n    st, ok := status.FromError(err)\n    if !ok { return false }\n    switch st.Code() {\n    case codes.Unavailable, codes.DeadlineExceeded, codes.Internal:\n        return true\n    }\n    return false\n}","tryCatchPattern":"// The client already retries with backoff; just observe connectivity.\nif conn.GetState() == connectivity.TransientFailure {\n    // log/metric, then wait; do not abort the connection.\n}","preventionTips":["Do not treat TRANSIENT_FAILURE as terminal; rely on the built-in backoff.","Implement server-side Health/Watch robustly (no panics mid-stream).","Stabilize the network/TLS path so streams are not reset."],"tags":["grpc","health-checking","connectivity","transient-failure","retry"],"analyzedSha":"03255a9237b6eb32710f6bc4f2de9a675b99fe36","analyzedAt":"2026-08-07T00:29:34.215Z","schemaVersion":2},"datasetVersion":"2026-08-07T03:17:09.362Z"}