grpc/grpc-go · warning

connection active but received health check RPC error: %v

Error message

connection active but received health check RPC error: %v

What it means

Produced by clientHealthCheck when the health-check Watch stream's RecvMsg returns a non-nil error whose code is not Unimplemented. The connection is marked TRANSIENT_FAILURE and the client backs off and retries. This is an expected, recoverable condition reported through the connectivity state machine rather than a fatal error.

Source

Thrown at health/client.go:104

		if err = s.SendMsg(&healthpb.HealthCheckRequest{Service: service}); err != nil && err != io.EOF {
			// Stream should have been closed, so we can safely continue to create a new stream.
			continue retryConnection
		}
		s.CloseSend()

		resp := new(healthpb.HealthCheckResponse)
		for {
			err = s.RecvMsg(resp)

			// Reports healthy for the LBing purposes if health check is not implemented in the server.
			if status.Code(err) == codes.Unimplemented {
				setConnectivityState(connectivity.Ready, nil)
				return err
			}

			// Reports unhealthy if server's Watch method gives an error other than UNIMPLEMENTED.
			if err != nil {
				setConnectivityState(connectivity.TransientFailure, fmt.Errorf("connection active but received health check RPC error: %v", err))
				continue retryConnection
			}

			// As a message has been received, removes the need for backoff for the next retry by resetting the try count.
			tryCnt = 0
			if resp.Status == healthpb.HealthCheckResponse_SERVING {
				setConnectivityState(connectivity.Ready, nil)
			} else {
				setConnectivityState(connectivity.TransientFailure, fmt.Errorf("connection active but health check failed. status=%s", resp.Status))
			}
		}
	}
}

View on GitHub (pinned to 03255a9237)

Solutions

1. Treat TRANSIENT_FAILURE as transient: the gRPC client already retries with exponential backoff, so no immediate action is needed.
2. Check server-side logs for the failing Health/Watch RPC to find the root cause.
3. Verify network stability and TLS configuration between client and server.
4. If persistent, confirm the server actually serves the gRPC Health service and the requested service name exists.

Example fix

// before: panicking on TRANSIENT_FAILURE
conn.WaitForStateChange(ctx, connectivity.TransientFailure)
panic("health failed")
// after: treat as transient, wait for READY
for conn.GetState() != connectivity.Ready {
    if !conn.WaitForStateChange(ctx, conn.GetState()) {
        return ctx.Err()
    }
}

// Distinguish health-check stream errors from other gRPC errors.
func isHealthCheckTransient(err error) bool {
    st, ok := status.FromError(err)
    if !ok { return false }
    switch st.Code() {
    case codes.Unavailable, codes.DeadlineExceeded, codes.Internal:
        return true
    }
    return false
}

// The client already retries with backoff; just observe connectivity.
if conn.GetState() == connectivity.TransientFailure {
    // log/metric, then wait; do not abort the connection.
}

When it happens

Trigger: Server returns UNAVAILABLE/INTERNAL on the Health/Watch RPC; the underlying transport breaks mid-stream (connection reset, TLS handshake failure, GOAWAY); server implements Health but errors the stream with a non-Unimplemented code.

Common situations: Server process restarting or crashing mid-RPC; network blip between client and server; misconfigured TLS causing intermittent failures; server that partially implements health checking; interop with a proxy that terminates long-lived streams.

Related errors

• connection active but health check failed. status=%s
• all SubConns are in TransientFailure
• token file access error
• empty token_exchange_service_uri in options
• required field SubjectTokenPath is not specified

AI-assisted analysis of grpc/grpc-go@03255a9237 (2026-08-07). Data as JSON: /api/errors/92e7a8d9063c5270. Report an issue: GitHub.