vitessio/vitess · error

tablet is not healthy. tablet: %v health record: %v

Error message

tablet is not healthy. tablet: %v health record: %v

What it means

While waiting for filtered (vreplication) replication to catch up, the streamed health record reports a non-empty HealthError, meaning the tablet itself considers itself unhealthy. The wait is aborted immediately because an unhealthy tablet's replication lag is meaningless. The error includes the full health record for diagnosis.

Source

Thrown at go/vt/wrangler/split.go:116

	// This is especially true for tests and automation where there is no pause of multiple seconds
	// between commands and the periodic healthcheck did not run again yet.
	if err := wr.TabletManagerClient().RunHealthCheck(ctx, tabletInfo.Tablet); err != nil {
		return fmt.Errorf("failed to run explicit healthcheck on tablet: %v err: %v", tabletInfo, err)
	}

	conn, err := tabletconn.GetDialer()(ctx, tabletInfo.Tablet, grpcclient.FailFast(false))
	if err != nil {
		return fmt.Errorf("cannot connect to tablet %v: %v", alias, err)
	}

	var lastSeenDelay time.Duration
	err = conn.StreamHealth(ctx, func(shr *querypb.StreamHealthResponse) error {
		stats := shr.RealtimeStats
		if stats == nil {
			return fmt.Errorf("health record does not include RealtimeStats message. tablet: %v health record: %v", alias, shr)
		}
		if stats.HealthError != "" {
			return fmt.Errorf("tablet is not healthy. tablet: %v health record: %v", alias, shr)
		}
		if stats.BinlogPlayersCount == 0 {
			return fmt.Errorf("no filtered replication running on tablet: %v health record: %v", alias, shr)
		}

		delaySecs := stats.FilteredReplicationLagSeconds
		lastSeenDelay = time.Duration(delaySecs) * time.Second
		if lastSeenDelay < 0 {
			return fmt.Errorf("last seen delay should never be negative. tablet: %v delay: %v", alias, lastSeenDelay)
		}
		if lastSeenDelay <= maxDelay {
			wr.Logger().Printf("Filtered replication on tablet: %v has caught up. Last seen delay: %.1f seconds\n", alias, lastSeenDelay.Seconds())
			return io.EOF
		}
		wr.Logger().Printf("Waiting for filtered replication to catch up on tablet: %v Last seen delay: %.1f seconds\n", alias, lastSeenDelay.Seconds())
		return nil
	})
	if err != nil {

View on GitHub (pinned to 01a25a7d17)

Solutions

1. Read the HealthError inside the printed health record to find the underlying MySQL/tablet problem
2. Fix the tablet health issue (restart replication with `vtctldclient ReplicationStart`, fix MySQL, free disk)
3. Re-run WaitForFilteredReplication once the tablet reports healthy
4. Check `vtctldclient GetTablet` and tablet /debug/vars for health status before retrying

Example fix

// before: waiting on an unhealthy tablet fails immediately
err := wr.WaitForFilteredReplication(ctx, alias, maxDelay)
// after: assert health first so the failure is caught at the right layer
hc, err := wr.VREngine().TabletHealth(ctx, alias)
if err != nil || hc.HealthError != "" {
	return fmt.Errorf("fix tablet health before waiting: %w", err)
}
err = wr.WaitForFilteredReplication(ctx, alias, maxDelay)

shr, err := getLatestHealth(ctx, wr, alias)
if err != nil || shr.GetRealtimeStats().HealthError != "" {
	return fmt.Errorf("tablet %s unhealthy: %v", alias, shr.GetRealtimeStats().GetHealthError())
}

func tabletHealthy(shr *querypb.StreamHealthResponse) bool {
	return shr.GetRealtimeStats() != nil && shr.GetRealtimeStats().HealthError == ""
}

if err := wr.WaitForFilteredReplication(ctx, alias, maxDelay); err != nil {
	if strings.Contains(err.Error(), "tablet is not healthy") {
		// parse health record, fix MySQL/replication, retry
	}
	return err
}

When it happens

Trigger: Calling WaitForFilteredReplication on a tablet whose RealtimeStats.HealthError is set — the tablet's underlying MySQL replica/health check is failing (e.g. replication stopped, MySQL down).

Common situations: Filtered replication lag monitoring during a MoveTables migration when the tablet's MySQL replica thread has stopped; disk full or MySQL restarted on the target tablet; health check failing due to MySQL auth or replication errors.

Related errors

• Last_SQL_Error: ${LastSQL_Error}, Last_IO_Error: ${LastIO_Er
• GetPreviousGTIDs: previous GTIDs not found
• ErrNoSemiSync
• no rpl_semi_sync_replica_status variable in mysql
• startPos.GTIDSet is wrong type - expected filePosGTID, got:

AI-assisted analysis of vitessio/vitess@01a25a7d17 (2026-09-01). Data as JSON: /api/errors/07a147873747cf37. Report an issue: GitHub.