grpc/grpc-go · error

pickfirst: health check failure: %v

Error message

pickfirst: health check failure: %v

What it means

pickfirst sets this as the picker's error when an established SubConn's health-check stream reports TransientFailure (see updateSubConnHealthState in pickfirst.go:797-827). It means the underlying transport came up, but the grpc.health.v1 HealthCheckResponse was not SERVING (or the health RPC itself failed), so the balancer treats the connection as unusable for picks. The %v is the SubConn's ConnectionError from the health producer. It surfaces to RPCs as status Unavailable.

Source

Thrown at balancer/pickfirst/pickfirst.go:817

	defer b.mu.Unlock()
	// Previously relevant SubConns can still callback with state updates.
	// To prevent pickers from returning these obsolete SubConns, this logic
	// is included to check if the current list of active SubConns includes
	// this SubConn.
	if !b.isActiveSCData(sd) {
		return
	}
	sd.effectiveState = state.ConnectivityState
	switch state.ConnectivityState {
	case connectivity.Ready:
		b.updateBalancerState(balancer.State{
			ConnectivityState: connectivity.Ready,
			Picker:            &picker{result: balancer.PickResult{SubConn: sd.subConn}},
		})
	case connectivity.TransientFailure:
		b.updateBalancerState(balancer.State{
			ConnectivityState: connectivity.TransientFailure,
			Picker:            &picker{err: fmt.Errorf("pickfirst: health check failure: %v", state.ConnectionError)},
		})
	case connectivity.Connecting:
		b.updateBalancerState(balancer.State{
			ConnectivityState: connectivity.Connecting,
			Picker:            &picker{err: balancer.ErrNoSubConnAvailable},
		})
	default:
		b.logger.Errorf("Got unexpected health update for SubConn %p: %v", state)
	}
}

// updateBalancerState stores the state reported to the channel and calls
// ClientConn.UpdateState(). As an optimization, it avoids sending duplicate
// updates to the channel.
func (b *pickfirstBalancer) updateBalancerState(newState balancer.State) {
	// In case of TransientFailures allow the picker to be updated to update
	// the connectivity error, in all other cases don't send duplicate state
	// updates.

View on GitHub (pinned to 03255a9237)

Solutions

1. On the server, register the health service and set SERVING: register health.NewServer(), then hs.SetServingStatus(healthCheckConfig.ServiceName, HealthCheckResponse_SERVING) once the app is ready.
2. Verify the healthCheckConfig.ServiceName in the client service config exactly matches the service name the server reports SERVING for (or use empty for overall health).
3. Wait for the connection to become READY and health SERVING before sending traffic, or treat status Unavailable from picks as transient and retry with backoff (the balancer auto-recovers and re-connects IDLE SubConns on TF).
4. Check server logs / channelz for the underlying ConnectionError in %v to see whether the health RPC failed or the server returned a non-SERVING status.

Example fix

// before: server starts serving but never reports health; client uses healthCheckConfig
s := grpc.NewServer()
pb.RegisterFooServer(s, foo)
// health never registered -> client picks fail with "pickfirst: health check failure"

// after
import healthpb "google.golang.org/grpc/health/grpc_health_v1"
import "google.golang.org/grpc/health"
hs := health.NewServer()
healthpb.RegisterHealthServer(s, hs)
hs.SetServingStatus("", healthpb.HealthCheckResponse_SERVING) // mark ready
hs.SetServingStatus("pkg.Foo", healthpb.HealthCheckResponse_SERVING)

// health-check readiness gate before sending traffic
func serving(ctx context.Context, conn *grpc.ClientConn, svc string) error {
    hc := healthpb.NewHealthClient(conn)
    r, err := hc.Check(ctx, &healthpb.HealthCheckRequest{Service: svc})
    if err != nil {
        return err
    }
    if r.Status != healthpb.HealthCheckResponse_SERVING {
        return fmt.Errorf("health not SERVING: %s", r.Status)
    }
    return nil
}

// RPCs fail with status Unavailable while health is down; retry transiently.
_, err := client.Call(ctx, req)
if st, ok := status.FromError(err); ok && st.Code() == codes.Unavailable {
    // balancer auto-recovers; back off and retry
}

When it happens

Trigger: Client service config has a non-empty healthCheckConfig.ServiceName; the server connection is READY but then the health server returns NOT_SERVING/SERVICE_UNKNOWN or the health-check RPC is cancelled/fails. At that point updateSubConnHealthState switches the picker to this error and every Pick() fails with it until health returns SERVING.

Common situations: Server forgot to call health.RegisterHealthServer / SetServingStatus, so the service is registered but never marked SERVING. Deployments that start traffic before the app is ready (health still NOT_SERVING). Mismatched healthCheckConfig.ServiceName vs the name the server reports under. Health-check RPC failing due to TLS/auth misconfig on an otherwise-open connection.

Related errors

• endpoints list is empty
• endpoints list contains no addresses
• randomsubsetting: json.Unmarshal failed for configuration: %
• randomsubsetting: SubsetSize must be greater than 0
• randomsubsetting: ChildPolicy must be specified

AI-assisted analysis of grpc/grpc-go@03255a9237 (2026-08-07). Data as JSON: /api/errors/b8f790d463cef3f2. Report an issue: GitHub.