vitessio/vitess · warning

vttablet error: %v

Error message

vttablet error: %v

What it means

When a vttablet reports a non-empty HealthError in its RealtimeStats, the healthcheck converts it into this 'vttablet error' and forces the tablet's serving state to false. It reflects an application-level health failure reported by the tablet itself (e.g. MySQL replication lag or liveness check failing), not a transport problem.

Source

Thrown at go/vt/discovery/tablet_health_check.go:177

// processResponse reads one health check response, and updates health
func (thc *tabletHealthCheck) processResponse(hc *HealthCheckImpl, shr *query.StreamHealthResponse) error {
	select {
	case <-thc.ctx.Done():
		return thc.ctx.Err()
	default:
	}

	// Check for invalid data, better than panicking.
	if shr.Target == nil || shr.RealtimeStats == nil {
		return fmt.Errorf("health stats is not valid: %v", shr)
	}

	// an app-level error from tablet, force serving state.
	var healthErr error
	serving := shr.Serving
	if shr.RealtimeStats.HealthError != "" {
		healthErr = fmt.Errorf("vttablet error: %v", shr.RealtimeStats.HealthError)
		serving = false
	}

	if shr.TabletAlias != nil && !proto.Equal(shr.TabletAlias, thc.Tablet.Alias) {
		// TabletAlias change means that the host:port has been taken over by another tablet
		// We cancel / exit the healthcheck for this tablet right away
		// With the next topo refresh we will get a new tablet with the new host/port
		return vterrors.New(vtrpc.Code_FAILED_PRECONDITION, fmt.Sprintf("health stats mismatch, tablet %+v alias does not match response alias %v", thc.Tablet, shr.TabletAlias))
	}

	prevTarget := thc.Target
	// check whether this is a trivial update so as to update healthy map
	trivialUpdate := thc.LastError == nil && thc.Serving && shr.RealtimeStats.HealthError == "" && shr.Serving &&
		prevTarget.TabletType != topodata.TabletType_PRIMARY && prevTarget.TabletType == shr.Target.TabletType && thc.isTrivialReplagChange(shr.RealtimeStats)
	thc.lastResponseTimestamp = time.Now()
	thc.Target = shr.Target
	thc.PrimaryTermStartTime = shr.PrimaryTermStartTimestamp
	thc.Stats = shr.RealtimeStats

View on GitHub (pinned to 01a25a7d17)

Solutions

1. Read the wrapped %v message — it contains vttablet's actual HealthError text and diagnose that condition.
2. Check replication status on the tablet's MySQL (SHOW REPLICA STATUS) for lag or broken replication.
3. Verify mysqld is running and responsive on the tablet host; restart if needed.
4. Once the underlying health condition clears, the tablet will resume serving automatically on the next healthy StreamHealthResponse.

hd, err := hc.GetTabletHealthByAlias(ctx, alias)
if err != nil && strings.Contains(err.Error(), "vttablet error") {
	// tablet reported unhealthy; retry or route traffic elsewhere
	return routeAround(alias)
}

When it happens

Trigger: processResponse receives a StreamHealthResponse whose RealtimeStats.HealthError is non-empty — vttablet's health check failed (e.g. MySQL down, replication lag above threshold, tablet overwhelmed).

Common situations: MySQL replica lagging beyond the lameduck/hi-jack threshold; mysqld restart on the tablet host; disk full or table locked causing health queries to fail; tablet freshly started before MySQL is ready.

Related errors

• invalid key:value pair
• Last_SQL_Error: ${LastSQL_Error}, Last_IO_Error: ${LastIO_Er
• GetPreviousGTIDs: previous GTIDs not found
• ErrNoSemiSync
• no rpl_semi_sync_replica_status variable in mysql

AI-assisted analysis of vitessio/vitess@01a25a7d17 (2026-09-01). Data as JSON: /api/errors/3298102edec38edf. Report an issue: GitHub.