{"record":{"id":"92e7a8d9063c5270","repo":"grpc/grpc-go","slug":"connection-active-but-received-health-check-rpc-er","errorCode":null,"errorMessage":"connection active but received health check RPC error: %v","messagePattern":"connection active but received health check RPC error: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"health/client.go","lineNumber":104,"sourceCode":"\t\tif err = s.SendMsg(&healthpb.HealthCheckRequest{Service: service}); err != nil && err != io.EOF {\n\t\t\t// Stream should have been closed, so we can safely continue to create a new stream.\n\t\t\tcontinue retryConnection\n\t\t}\n\t\ts.CloseSend()\n\n\t\tresp := new(healthpb.HealthCheckResponse)\n\t\tfor {\n\t\t\terr = s.RecvMsg(resp)\n\n\t\t\t// Reports healthy for the LBing purposes if health check is not implemented in the server.\n\t\t\tif status.Code(err) == codes.Unimplemented {\n\t\t\t\tsetConnectivityState(connectivity.Ready, nil)\n\t\t\t\treturn err\n\t\t\t}\n\n\t\t\t// Reports unhealthy if server's Watch method gives an error other than UNIMPLEMENTED.\n\t\t\tif err != nil {\n\t\t\t\tsetConnectivityState(connectivity.TransientFailure, fmt.Errorf(\"connection active but received health check RPC error: %v\", err))\n\t\t\t\tcontinue retryConnection\n\t\t\t}\n\n\t\t\t// As a message has been received, removes the need for backoff for the next retry by resetting the try count.\n\t\t\ttryCnt = 0\n\t\t\tif resp.Status == healthpb.HealthCheckResponse_SERVING {\n\t\t\t\tsetConnectivityState(connectivity.Ready, nil)\n\t\t\t} else {\n\t\t\t\tsetConnectivityState(connectivity.TransientFailure, fmt.Errorf(\"connection active but health check failed. status=%s\", resp.Status))\n\t\t\t}\n\t\t}\n\t}\n}\n","sourceCodeStart":86,"sourceCodeEnd":118,"githubUrl":"https://github.com/grpc/grpc-go/blob/0c51461d27177d997e14c642fe18c11668fc09a3/health/client.go#L86-L118","documentation":"The gRPC client health-check watcher received an RPC error (other than UNIMPLEMENTED) on an open connection. Per the grpc health protocol, this sets the subchannel to TransientFailure and triggers backoff/retry; the wrapped %v is the stream RecvMsg error. This is the health-check state machine reporting a server-side Watch failure, not necessarily a transport failure.","triggerScenarios":"Server's Health/Watch RPC returned an error status: CANCELLED (server shutdown), DEADLINE_EXCEEDED, PERMISSION_DENIED, INTERNAL, or the stream was reset. Emitted at health/client.go:104 during clientHealthCheck.","commonSituations":"Server process restarting or draining; server has a buggy Health service implementation that closes the stream early; interceptor/proxy resetting HTTP/2 streams; auth/z policy rejecting the Watch call; server panic in the Health handler; load balancer timing out long-lived streams.","solutions":["Inspect the inner %v: CANCELLED/UNAVAILABLE usually means server restart; let the client's built-in backoff retry.","If PERMISSION_DENIED, fix the auth policy on the server's Health service.","If the stream resets repeatedly, check server logs for panics in the Health implementation and ensure the service is registered with the gRPC server.","Stabilize long-lived streams: raise HTTP/2 keepalive timeouts on intermediaries, or use keepalive.EnforcementPolicy on the server.","Confirm the server implements grpc.health.v1.Health (UNIMPLEMENTED is handled separately and is non-fatal)."],"exampleFix":"// before: server registers no Health service and Watch returns INTERNAL on a bad handler\nsrv := grpc.NewServer()\n\n// after\nimport healthpb \"google.golang.org/grpc/health\"\nimport healthsvc \"google.golang.org/grpc/health/grpc_health_v1\"\nhs := healthpb.NewServer()\nhs.SetServingStatus(\"\", healthpb.HealthCheckResponse_SERVING)\nhealthsvc.RegisterHealthServer(srv, hs)","handlingStrategy":"retry","validationCode":"// Health is built into grpc-go's connection; pre-flight by checking the server implements Health:\nimport healthpb \"google.golang.org/grpc/health/grpc_health_v1\"\n\nhc := healthpb.NewHealthClient(conn)\nresp, err := hc.Check(ctx, &healthpb.HealthCheckRequest{Service: svc})\nif err != nil { return fmt.Errorf(\"server Health not usable: %w\", err) }\nif resp.Status != healthpb.HealthCheckResponse_SERVING { return fmt.Errorf(\"server not serving: %s\", resp.Status) }","typeGuard":null,"tryCatchPattern":"// The health-check state machine already retries with backoff; surface non-fatal failures via connectivity callbacks.\nconn.WaitForStateChange(ctx, connectivity.Ready) // or subscribe to state updates\n// Treat TransientFailure as a signal to shed load; rely on the built-in retry, do not crash.","preventionTips":["Implement and register the grpc.health.v1.Health service on every server.","Use keepalive.EnforcementPolicy and raise HTTP/2 stream timeouts on intermediaries.","Watch subchannel connectivity states instead of raw health errors."],"tags":["health-check","client","load-balancing","connection"],"backgroundTag":null,"analyzedSha":"0c51461d27177d997e14c642fe18c11668fc09a3","analyzedAt":"2026-08-11T14:49:15.055Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}