grpc/grpc-go · warning

connection active but health check failed. status=

Error message

connection active but health check failed. status=%s

What it means

The health-check Watch RPC succeeded and returned a HealthCheckResponse whose Status is not SERVING (i.e. NOT_SERVING, SERVICE_UNKNOWN, or UNKNOWN). The connection is up but the server is explicitly reporting unhealthy, so the client puts the subchannel into TransientFailure. Emitted at health/client.go:113.

Solutions

1. On the server, ensure healthServer.SetServingStatus(<service>, SERVING) is called once the service is ready; for whole-server health use the empty service name "".
2. Match the service name on both sides: client config service_name must equal what the server sets.
3. If NOT_SERVING is expected during shutdown, treat it as a drain signal in your load balancer; the client will retry when status flips back.
4. For SERVICE_UNKNOWN, register a Health server and explicitly set status for the requested service, or have the client use "" for overall server health.
5. Investigate downstream dependencies if the app is flipping to NOT_SERVING unexpectedly.

Example fix

// before: server never sets serving status
hs := healthpb.NewServer()
healthsvc.RegisterHealthServer(srv, hs)

// after
hs := healthpb.NewServer()
hs.SetServingStatus("", healthpb.HealthCheckResponse_SERVING)
hs.SetServingStatus("foo.Bar", healthpb.HealthCheckResponse_SERVING)
healthsvc.RegisterHealthServer(srv, hs)

// Verify the server reports SERVING for the service the client will request.
import healthpb "google.golang.org/grpc/health/grpc_health_v1"

resp, err := healthpb.NewHealthClient(conn).Check(ctx, &healthpb.HealthCheckRequest{Service: svc})
if err != nil { return err }
if resp.Status != healthpb.HealthCheckResponse_SERVING {
    return fmt.Errorf("server reports %s for %q; do not send traffic", resp.Status, svc)
}

// The client already maps non-SERVING to TransientFailure. In your app, fall back to another replica or shed load.
if state := conn.GetState(); state == connectivity.TransientFailure {
    // route around this backend; retry will resume when status flips to SERVING
}

When it happens

Trigger: Server called healthServer.SetServingStatus(svc, NOT_SERVING) (e.g. during shutdown or dependency failure), or returned SERVICE_UNKNOWN because the requested service name is not tracked. The %s prints the enum value (NOT_SERVING / SERVICE_UNKNOWN / UNKNOWN).

Common situations: Server intentionally marks itself NOT_SERVING during graceful shutdown or readiness probes failing; service name mismatch (client asks for "foo.Bar" but server only sets status for "" or a different name) yielding SERVICE_UNKNOWN; a dependency (DB) goes down so the app flips its own health; rolling deploy flipping status.

Related errors

• connection active but received health check RPC error
• duplicated name
• endpoints list contains no addresses
• endpoints list is empty
• last resolver error

AI-assisted analysis of grpc/grpc-go@0c51461d27 (2026-08-11). Data as JSON: /api/errors/75b9407c6f2742af. Report an issue: GitHub.

Appendix: source

Thrown at health/client.go:113

			// Reports healthy for the LBing purposes if health check is not implemented in the server.
			if status.Code(err) == codes.Unimplemented {
				setConnectivityState(connectivity.Ready, nil)
				return err
			}

			// Reports unhealthy if server's Watch method gives an error other than UNIMPLEMENTED.
			if err != nil {
				setConnectivityState(connectivity.TransientFailure, fmt.Errorf("connection active but received health check RPC error: %v", err))
				continue retryConnection
			}

			// As a message has been received, removes the need for backoff for the next retry by resetting the try count.
			tryCnt = 0
			if resp.Status == healthpb.HealthCheckResponse_SERVING {
				setConnectivityState(connectivity.Ready, nil)
			} else {
				setConnectivityState(connectivity.TransientFailure, fmt.Errorf("connection active but health check failed. status=%s", resp.Status))
			}
		}
	}
}

View on GitHub (pinned to 0c51461d27)