vitessio/vitess · error

health stats is not valid: %v

Error message

health stats is not valid: %v

What it means

processResponse validates every streaming health check response from a vttablet. If the StreamHealthResponse is missing its Target or RealtimeStats, the data is considered malformed and this error is returned instead of risking a nil-pointer panic, and the tablet's health state is not updated.

Source

Thrown at go/vt/discovery/tablet_health_check.go:170

			return nil
		}
		thc.Conn = conn
		thc.LastError = nil
	}
	return thc.Conn
}

// processResponse reads one health check response, and updates health
func (thc *tabletHealthCheck) processResponse(hc *HealthCheckImpl, shr *query.StreamHealthResponse) error {
	select {
	case <-thc.ctx.Done():
		return thc.ctx.Err()
	default:
	}

	// Check for invalid data, better than panicking.
	if shr.Target == nil || shr.RealtimeStats == nil {
		return fmt.Errorf("health stats is not valid: %v", shr)
	}

	// an app-level error from tablet, force serving state.
	var healthErr error
	serving := shr.Serving
	if shr.RealtimeStats.HealthError != "" {
		healthErr = fmt.Errorf("vttablet error: %v", shr.RealtimeStats.HealthError)
		serving = false
	}

	if shr.TabletAlias != nil && !proto.Equal(shr.TabletAlias, thc.Tablet.Alias) {
		// TabletAlias change means that the host:port has been taken over by another tablet
		// We cancel / exit the healthcheck for this tablet right away
		// With the next topo refresh we will get a new tablet with the new host/port
		return vterrors.New(vtrpc.Code_FAILED_PRECONDITION, fmt.Sprintf("health stats mismatch, tablet %+v alias does not match response alias %v", thc.Tablet, shr.TabletAlias))
	}

	prevTarget := thc.Target

View on GitHub (pinned to 01a25a7d17)

Solutions

1. Check the responding vttablet's version and upgrade it to match the gateway/cluster version.
2. Look at vttablet logs around the time of the malformed response for startup or crash issues.
3. Restart the affected vttablet if it persistently sends incomplete StreamHealthResponses.
4. Verify network path (no truncating proxies/LBs) between healthcheck and tablet.

func validHealthResponse(shr *querypb.StreamHealthResponse) bool {
	return shr != nil && shr.Target != nil && shr.RealtimeStats != nil
}

When it happens

Trigger: A vttablet returns a StreamHealthResponse with a nil Target or nil RealtimeStats — typically an old/mismatched vttablet version, a tablet starting up, or a corrupted/partial RPC response.

Common situations: Mixed-version clusters (new gateways talking to older tablets); vttablet crash/restart mid-health-stream producing empty responses; proxy or networking layer truncating gRPC messages.

Related errors

• mysqld >= 8.0.21 required to disable the redo log
• failed to load static auth plugin. Plugin configured but grp
• gRPCVtctldClient in a SHUTDOWN state
• gRPC connection wait time exceeded
• ErrStreamClosed

AI-assisted analysis of vitessio/vitess@01a25a7d17 (2026-09-01). Data as JSON: /api/errors/e8650b5121a8875b. Report an issue: GitHub.