grpc/grpc-go · warning

connection active but health check failed. status=%s

Error message

connection active but health check failed. status=%s

What it means

Produced by clientHealthCheck when the server's Health/Watch RPC returns a HealthCheckResponse with a status other than SERVING (i.e. NOT_SERVING, SERVICE_UNKNOWN, or UNKNOWN). The connection is set to TRANSIENT_FAILURE even though the transport is healthy, because the application layer reports the service is not ready.

Source

Thrown at health/client.go:113

			// Reports healthy for the LBing purposes if health check is not implemented in the server.
			if status.Code(err) == codes.Unimplemented {
				setConnectivityState(connectivity.Ready, nil)
				return err
			}

			// Reports unhealthy if server's Watch method gives an error other than UNIMPLEMENTED.
			if err != nil {
				setConnectivityState(connectivity.TransientFailure, fmt.Errorf("connection active but received health check RPC error: %v", err))
				continue retryConnection
			}

			// As a message has been received, removes the need for backoff for the next retry by resetting the try count.
			tryCnt = 0
			if resp.Status == healthpb.HealthCheckResponse_SERVING {
				setConnectivityState(connectivity.Ready, nil)
			} else {
				setConnectivityState(connectivity.TransientFailure, fmt.Errorf("connection active but health check failed. status=%s", resp.Status))
			}
		}
	}
}

View on GitHub (pinned to 03255a9237)

Solutions

1. Confirm the server is intentionally non-serving (check its health.SetServingStatus call sites).
2. Verify the service name in the health check request matches a name the server registered.
3. Wait for the server to transition back to SERVING; the client auto-recovers.
4. During deploys, ensure readiness probes align with SetServingStatus(true).

Example fix

// server side: ensure you flip serving status correctly
// before
healthServer.SetServingStatus("my.Service", healthpb.HealthCheckResponse_NOT_SERVING)
// after, once dependencies are ready
healthServer.SetServingStatus("my.Service", healthpb.HealthCheckResponse_SERVING)

// Inspect a HealthCheckResponse to decide whether to keep traffic away.
func isServing(resp *healthpb.HealthCheckResponse) bool {
    return resp.GetStatus() == healthpb.HealthCheckResponse_SERVING
}

// On the client, route around a non-serving backend via the LB policy.
resp, err := healthClient.Check(ctx, &healthpb.HealthCheckRequest{Service: svc})
if err != nil || !isServing(resp) {
    // pick another backend / shed load
}

When it happens

Trigger: Server explicitly reports NOT_SERVING during shutdown or dependency failure; requesting a health check for a service name the server does not know (SERVICE_UNKNOWN); server-side health status set to non-serving by the application's SetServingStatus call.

Common situations: Rolling deploy where the new pod reports NOT_SERVING until ready; dependency (DB, downstream service) outage causing the server to mark itself unhealthy; querying health for an unregistered service name; graceful shutdown signaling.

Related errors

• connection active but received health check RPC error: %v
• endpoints list is empty
• endpoints list contains no addresses
• pickfirst: health check failure: %v
• randomsubsetting: json.Unmarshal failed for configuration: %

AI-assisted analysis of grpc/grpc-go@03255a9237 (2026-08-07). Data as JSON: /api/errors/75b9407c6f2742af. Report an issue: GitHub.