vitessio/vitess · error

health check failed on %s

Error message

health check failed on %s

What it means

The VStreamer streams tablet health for each source tablet via StreamHealth; when a response has no RealtimeStats or Target (or shr is nil) it produces 'health check failed on %s'. This signals the tablet's health stream returned an unusable/malformed response, so the vstream logs it, wraps the error as 'error streaming tablet health from %s' and sends it to errCh, causing the vstream to restart on that tablet.

Source

Thrown at go/vt/vtgate/vstream_manager.go:733

			Shard:      sgtid.Shard,
			TabletType: vs.tabletType,
			Cell:       vs.vsm.cell,
		}
		tabletConn, err := vs.vsm.resolver.GetGateway().QueryServiceByAlias(ctx, tablet.Alias, target)
		if err != nil {
			log.Error(err.Error())
			return vterrors.Wrapf(err, "failed to get tablet connection to %s", tabletAliasString)
		}

		errCh := make(chan error, 1)
		go func() {
			_ = tabletConn.StreamHealth(ctx, func(shr *querypb.StreamHealthResponse) error {
				var err error
				switch {
				case ctx.Err() != nil:
					err = vterrors.Wrapf(ctx.Err(), "context ended while streaming tablet health from %s", tabletAliasString)
				case shr == nil || shr.RealtimeStats == nil || shr.Target == nil:
					err = fmt.Errorf("health check failed on %s", tabletAliasString)
				case vs.tabletType != shr.Target.TabletType:
					err = fmt.Errorf("tablet %s type has changed from %s to %s, restarting vstream",
						topoproto.TabletAliasString(tablet.Alias), vs.tabletType, shr.Target.TabletType)
				case shr.RealtimeStats.HealthError != "":
					err = fmt.Errorf("tablet %s is no longer healthy: %s, restarting vstream",
						topoproto.TabletAliasString(tablet.Alias), shr.RealtimeStats.HealthError)
				case shr.RealtimeStats.ReplicationLagSeconds > uint32(discovery.GetLowReplicationLag().Seconds()):
					err = fmt.Errorf("tablet %s has a replication lag of %d seconds which is beyond the value provided in --discovery_low_replication_lag of %s so the tablet is no longer considered healthy, restarting vstream",
						topoproto.TabletAliasString(tablet.Alias), shr.RealtimeStats.ReplicationLagSeconds, discovery.GetLowReplicationLag())
				}
				if err != nil {
					log.Warn(fmt.Sprintf("Tablet state changed: %s, attempting to restart", err))
					err = vterrors.Wrapf(err, "error streaming tablet health from %s", tabletAliasString)
					errCh <- err
					return err
				}
				return nil
			})

View on GitHub (pinned to 01a25a7d17)

Solutions

1. Check the source tablet's logs and health (vtctldclient GetTablets / tablet web UI 15100/debug vars).
2. Confirm the tablet is RUNNING and serving the expected tablet type; restart the tablet if wedged.
3. Verify tablet and vtgate/vtctld versions are compatible (upgrade older tablets).
4. Check network stability between vtgate and the tablet's gRPC port.
5. VStream will attempt to restart on a healthy tablet — if persistent, rebuild the tablet or re-init replication.

Example fix

// before: tablet down during vstream
$ vtctldclient PlannedReparentShard  # reparent failed, tablet stale
// after: restore tablet health first
$ vtctldclient RebuildKeyspaceGraph && restart vittablet; then retry MoveTables

// before starting the workflow, check tablet health:
// vtctldclient GetTablets --keyspace ks → all sources must be SERVING
// curl http://<tablet>:15100/healthz  → must return OK

errCh := make(chan error, 1)
go func() { errCh <- startVStream() }()
select {
case err := <-errCh:
    if strings.Contains(err.Error(), "health check failed") {
        // wait for tablet recovery then restart the stream
        time.Sleep(backoff)
        retry()
    }
}

When it happens

Trigger: A tablet closes/initializes the health stream without sending a well-formed StreamHealthResponse (nil response, nil RealtimeStats, or nil Target) while a VReplication workflow (MoveTables, Reshard, etc.) streams from it.

Common situations: Tablet restarting or being taken down during migration; network interruption truncating health stream; tablet serving a different target than expected right after reparent; version mismatch where older tablets don't populate fields.

Related errors

• tablet %s is no longer healthy: %s, restarting vstream
• vstream ended
• could not delete target streams: %v
• could not restart source streams: %v
• tablet %s type has changed from %s to %s, restarting vstream

AI-assisted analysis of vitessio/vitess@01a25a7d17 (2026-09-01). Data as JSON: /api/errors/efc96f67a7bed5db. Report an issue: GitHub.