vitessio/vitess · error

health record does not include RealtimeStats message. tablet

Error message

health record does not include RealtimeStats message. tablet: %v health record: %v

What it means

During WaitForFilteredReplication, the wrangler streams health records from a tablet via StreamHealth and requires each response to carry a RealtimeStats message. If a StreamHealthResponse arrives with RealtimeStats == nil, there is no health data to evaluate, so the callback aborts with this error. This indicates the tablet is sending health streams but omitting the stats payload, typically a tablet-side anomaly or a very early/terminal stream message.

Source

Thrown at go/vt/wrangler/split.go:113

	}

	// Always run an explicit healthcheck first to make sure we don't see any outdated values.
	// This is especially true for tests and automation where there is no pause of multiple seconds
	// between commands and the periodic healthcheck did not run again yet.
	if err := wr.TabletManagerClient().RunHealthCheck(ctx, tabletInfo.Tablet); err != nil {
		return fmt.Errorf("failed to run explicit healthcheck on tablet: %v err: %v", tabletInfo, err)
	}

	conn, err := tabletconn.GetDialer()(ctx, tabletInfo.Tablet, grpcclient.FailFast(false))
	if err != nil {
		return fmt.Errorf("cannot connect to tablet %v: %v", alias, err)
	}

	var lastSeenDelay time.Duration
	err = conn.StreamHealth(ctx, func(shr *querypb.StreamHealthResponse) error {
		stats := shr.RealtimeStats
		if stats == nil {
			return fmt.Errorf("health record does not include RealtimeStats message. tablet: %v health record: %v", alias, shr)
		}
		if stats.HealthError != "" {
			return fmt.Errorf("tablet is not healthy. tablet: %v health record: %v", alias, shr)
		}
		if stats.BinlogPlayersCount == 0 {
			return fmt.Errorf("no filtered replication running on tablet: %v health record: %v", alias, shr)
		}

		delaySecs := stats.FilteredReplicationLagSeconds
		lastSeenDelay = time.Duration(delaySecs) * time.Second
		if lastSeenDelay < 0 {
			return fmt.Errorf("last seen delay should never be negative. tablet: %v delay: %v", alias, lastSeenDelay)
		}
		if lastSeenDelay <= maxDelay {
			wr.Logger().Printf("Filtered replication on tablet: %v has caught up. Last seen delay: %.1f seconds\n", alias, lastSeenDelay.Seconds())
			return io.EOF
		}
		wr.Logger().Printf("Waiting for filtered replication to catch up on tablet: %v Last seen delay: %.1f seconds\n", alias, lastSeenDelay.Seconds())

View on GitHub (pinned to 01a25a7d17)

Solutions

1. Check the tablet's version and restart vttablet so it emits RealtimeStats in StreamHealth responses
2. Verify with `vtctldclient GetTablet <alias>` / health check that the tablet is serving and healthy
3. Retry WaitForFilteredReplication after confirming the tablet's health stream works (vtctl HealthCheck or tablet debug vars)
4. If the tablet is being decommissioned mid-migration, wait for it to stabilize or exclude it from the workflow

Example fix

// before: blindly waiting for filtered replication on a possibly misbehaving tablet
wr.WaitForFilteredReplication(ctx, alias, maxDelay)
// after: pre-check tablet health before streaming
if err := checkTabletHealth(ctx, wr, alias); err != nil {
	return fmt.Errorf("tablet %s unhealthy before wait: %w", alias, err)
}
wr.WaitForFilteredReplication(ctx, alias, maxDelay)

hc, err := getTabletHealth(ctx, wr, alias)
if err != nil || hc.RealtimeStats == nil {
	return fmt.Errorf("tablet %s does not emit RealtimeStats; fix vttablet before waiting", alias)
}

func hasRealtimeStats(shr *querypb.StreamHealthResponse) bool {
	return shr != nil && shr.RealtimeStats != nil
}

if err := wr.WaitForFilteredReplication(ctx, alias, maxDelay); err != nil {
	if strings.Contains(err.Error(), "does not include RealtimeStats") {
		// restart/upgrade vttablet, then retry
	}
	return err
}

When it happens

Trigger: Calling wrangler.WaitForFilteredReplication (via vtctlcommand WaitForFilteredReplication / commandWaitForFilteredReplication) against a tablet whose StreamHealthResponse frames lack RealtimeStats, e.g. a tablet running an old binary or one whose health stream closes/misses the stats message.

Common situations: Mixed-version clusters where a source tablet streams health without RealtimeStats populated; tablet health stream terminating immediately after connect; vttablet misconfiguration disabling realtime stats reporting.

Related errors

• stream error @ (including the GTID we failed to process) %v:
• tablet is not healthy. tablet: %v health record: %v
• last seen delay should never be negative. tablet: %v delay:
• could not stream health records from tablet: %v err: %v
• invalid key:value pair

AI-assisted analysis of vitessio/vitess@01a25a7d17 (2026-09-01). Data as JSON: /api/errors/a7ffc807abe95110. Report an issue: GitHub.