vitessio/vitess · error

tablet %s is no longer healthy: %s, restarting vstream

Error message

tablet %s is no longer healthy: %s, restarting vstream

What it means

The tablet's health stream reports RealtimeStats.HealthError (the tablet itself says it is unhealthy, e.g. MySQL down, replication broken), so the VStreamer errors 'tablet %s is no longer healthy: %s, restarting vstream'. The inner message comes from the tablet's health check (mysql ping, replication status); VTGate wraps it and restarts streaming elsewhere.

Source

Thrown at go/vt/vtgate/vstream_manager.go:738

		if err != nil {
			log.Error(err.Error())
			return vterrors.Wrapf(err, "failed to get tablet connection to %s", tabletAliasString)
		}

		errCh := make(chan error, 1)
		go func() {
			_ = tabletConn.StreamHealth(ctx, func(shr *querypb.StreamHealthResponse) error {
				var err error
				switch {
				case ctx.Err() != nil:
					err = vterrors.Wrapf(ctx.Err(), "context ended while streaming tablet health from %s", tabletAliasString)
				case shr == nil || shr.RealtimeStats == nil || shr.Target == nil:
					err = fmt.Errorf("health check failed on %s", tabletAliasString)
				case vs.tabletType != shr.Target.TabletType:
					err = fmt.Errorf("tablet %s type has changed from %s to %s, restarting vstream",
						topoproto.TabletAliasString(tablet.Alias), vs.tabletType, shr.Target.TabletType)
				case shr.RealtimeStats.HealthError != "":
					err = fmt.Errorf("tablet %s is no longer healthy: %s, restarting vstream",
						topoproto.TabletAliasString(tablet.Alias), shr.RealtimeStats.HealthError)
				case shr.RealtimeStats.ReplicationLagSeconds > uint32(discovery.GetLowReplicationLag().Seconds()):
					err = fmt.Errorf("tablet %s has a replication lag of %d seconds which is beyond the value provided in --discovery_low_replication_lag of %s so the tablet is no longer considered healthy, restarting vstream",
						topoproto.TabletAliasString(tablet.Alias), shr.RealtimeStats.ReplicationLagSeconds, discovery.GetLowReplicationLag())
				}
				if err != nil {
					log.Warn(fmt.Sprintf("Tablet state changed: %s, attempting to restart", err))
					err = vterrors.Wrapf(err, "error streaming tablet health from %s", tabletAliasString)
					errCh <- err
					return err
				}
				return nil
			})
		}()

		var options *binlogdatapb.VStreamOptions
		const SidecarDBHeartbeatTableName = "heartbeat"
		if vs.flags.GetStreamKeyspaceHeartbeats() {

View on GitHub (pinned to 01a25a7d17)

Solutions

1. Read the inner HealthError text — fix the underlying tablet/MySQL issue (restart mysqld, fix replication, free disk).
2. Confirm tablet health endpoint shows healthy (localhost:15100/healthz).
3. Check replication status on the tablet (SHOW REPLICA STATUS) and repair if broken.
4. Let the vstream restart on a healthy tablet; verify workflow progress resumes (VReplication workflow status).
5. If one tablet repeatedly reports HealthError, investigate hardware/OS or take it out of the serving pool.

Example fix

// before
ERROR 2002 mysqld down → HealthError="dial tcp ... connection refused"
// after
systemctl restart mysqld; vtctldclient SetWritable alias true; workflow auto-restarts

// pre-check tablet health before streaming:
// curl http://<tablet>:15100/healthz → OK
// SHOW REPLICA STATUS on source → no Last_Error

err := streamHealth(ctx)
if err != nil && strings.Contains(err.Error(), "no longer healthy") {
    // inner HealthError tells the root cause; fix then retry
    log.Warn("tablet unhealthy", slog.Any("error", err))
    return retryWithBackoff(ctx)
}

When it happens

Trigger: RealtimeStats.HealthError becomes non-empty on the streaming tablet — MySQL restart, disk full, replication stopped, semi-sync timeout — during any VReplication workflow.

Common situations: Source MySQL crash or OOM-kill mid-copy; replica lagging/broken replication; tablet marked not serving after maintenance; storage failures on the source host.

Related errors

• health check failed on %s
• vstream ended
• partial row image encountered: ensure binlog_row_image is se
• can't get charset to request binlog stream: %v
• error in processing binlog event %v

AI-assisted analysis of vitessio/vitess@01a25a7d17 (2026-09-01). Data as JSON: /api/errors/64a4168e73355d59. Report an issue: GitHub.