{"record":{"id":"b8f790d463cef3f2","repo":"grpc/grpc-go","slug":"pickfirst-health-check-failure-v","errorCode":null,"errorMessage":"pickfirst: health check failure: %v","messagePattern":"pickfirst: health check failure: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"balancer/pickfirst/pickfirst.go","lineNumber":817,"sourceCode":"\tdefer b.mu.Unlock()\n\t// Previously relevant SubConns can still callback with state updates.\n\t// To prevent pickers from returning these obsolete SubConns, this logic\n\t// is included to check if the current list of active SubConns includes\n\t// this SubConn.\n\tif !b.isActiveSCData(sd) {\n\t\treturn\n\t}\n\tsd.effectiveState = state.ConnectivityState\n\tswitch state.ConnectivityState {\n\tcase connectivity.Ready:\n\t\tb.updateBalancerState(balancer.State{\n\t\t\tConnectivityState: connectivity.Ready,\n\t\t\tPicker:            &picker{result: balancer.PickResult{SubConn: sd.subConn}},\n\t\t})\n\tcase connectivity.TransientFailure:\n\t\tb.updateBalancerState(balancer.State{\n\t\t\tConnectivityState: connectivity.TransientFailure,\n\t\t\tPicker:            &picker{err: fmt.Errorf(\"pickfirst: health check failure: %v\", state.ConnectionError)},\n\t\t})\n\tcase connectivity.Connecting:\n\t\tb.updateBalancerState(balancer.State{\n\t\t\tConnectivityState: connectivity.Connecting,\n\t\t\tPicker:            &picker{err: balancer.ErrNoSubConnAvailable},\n\t\t})\n\tdefault:\n\t\tb.logger.Errorf(\"Got unexpected health update for SubConn %p: %v\", state)\n\t}\n}\n\n// updateBalancerState stores the state reported to the channel and calls\n// ClientConn.UpdateState(). As an optimization, it avoids sending duplicate\n// updates to the channel.\nfunc (b *pickfirstBalancer) updateBalancerState(newState balancer.State) {\n\t// In case of TransientFailures allow the picker to be updated to update\n\t// the connectivity error, in all other cases don't send duplicate state\n\t// updates.","sourceCodeStart":799,"sourceCodeEnd":835,"githubUrl":"https://github.com/grpc/grpc-go/blob/03255a9237b6eb32710f6bc4f2de9a675b99fe36/balancer/pickfirst/pickfirst.go#L799-L835","documentation":"pickfirst sets this as the picker's error when an established SubConn's health-check stream reports TransientFailure (see updateSubConnHealthState in pickfirst.go:797-827). It means the underlying transport came up, but the grpc.health.v1 HealthCheckResponse was not SERVING (or the health RPC itself failed), so the balancer treats the connection as unusable for picks. The %v is the SubConn's ConnectionError from the health producer. It surfaces to RPCs as status Unavailable.","triggerScenarios":"Client service config has a non-empty healthCheckConfig.ServiceName; the server connection is READY but then the health server returns NOT_SERVING/SERVICE_UNKNOWN or the health-check RPC is cancelled/fails. At that point updateSubConnHealthState switches the picker to this error and every Pick() fails with it until health returns SERVING.","commonSituations":"Server forgot to call health.RegisterHealthServer / SetServingStatus, so the service is registered but never marked SERVING. Deployments that start traffic before the app is ready (health still NOT_SERVING). Mismatched healthCheckConfig.ServiceName vs the name the server reports under. Health-check RPC failing due to TLS/auth misconfig on an otherwise-open connection.","solutions":["On the server, register the health service and set SERVING: register health.NewServer(), then hs.SetServingStatus(healthCheckConfig.ServiceName, HealthCheckResponse_SERVING) once the app is ready.","Verify the healthCheckConfig.ServiceName in the client service config exactly matches the service name the server reports SERVING for (or use empty for overall health).","Wait for the connection to become READY and health SERVING before sending traffic, or treat status Unavailable from picks as transient and retry with backoff (the balancer auto-recovers and re-connects IDLE SubConns on TF).","Check server logs / channelz for the underlying ConnectionError in %v to see whether the health RPC failed or the server returned a non-SERVING status."],"exampleFix":"// before: server starts serving but never reports health; client uses healthCheckConfig\ns := grpc.NewServer()\npb.RegisterFooServer(s, foo)\n// health never registered -> client picks fail with \"pickfirst: health check failure\"\n\n// after\nimport healthpb \"google.golang.org/grpc/health/grpc_health_v1\"\nimport \"google.golang.org/grpc/health\"\nhs := health.NewServer()\nhealthpb.RegisterHealthServer(s, hs)\nhs.SetServingStatus(\"\", healthpb.HealthCheckResponse_SERVING) // mark ready\nhs.SetServingStatus(\"pkg.Foo\", healthpb.HealthCheckResponse_SERVING)","handlingStrategy":"retry","validationCode":null,"typeGuard":"// health-check readiness gate before sending traffic\nfunc serving(ctx context.Context, conn *grpc.ClientConn, svc string) error {\n    hc := healthpb.NewHealthClient(conn)\n    r, err := hc.Check(ctx, &healthpb.HealthCheckRequest{Service: svc})\n    if err != nil {\n        return err\n    }\n    if r.Status != healthpb.HealthCheckResponse_SERVING {\n        return fmt.Errorf(\"health not SERVING: %s\", r.Status)\n    }\n    return nil\n}","tryCatchPattern":"// RPCs fail with status Unavailable while health is down; retry transiently.\n_, err := client.Call(ctx, req)\nif st, ok := status.FromError(err); ok && st.Code() == codes.Unavailable {\n    // balancer auto-recovers; back off and retry\n}","preventionTips":["On the server, always register health.NewServer() and call SetServingStatus(SERVING) only after the app is ready to serve.","Gate traffic on a health check (SERVING) before sending RPCs, especially at startup/deploy.","Keep healthCheckConfig.ServiceName aligned with the name the server reports.","Treat Unavailable from picks as transient and retry with backoff rather than failing hard."],"tags":["go","grpc","load-balancing","pick-first","health-check","connection"],"analyzedSha":"03255a9237b6eb32710f6bc4f2de9a675b99fe36","analyzedAt":"2026-08-07T00:29:34.215Z","schemaVersion":2},"datasetVersion":"2026-08-07T03:17:09.362Z"}