{"record":{"id":"64a4168e73355d59","repo":"vitessio/vitess","slug":"tablet-s-is-no-longer-healthy-s-restarting-vst","errorCode":null,"errorMessage":"tablet %s is no longer healthy: %s, restarting vstream","messagePattern":"tablet (.+?) is no longer healthy: (.+?), restarting vstream","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"go/vt/vtgate/vstream_manager.go","lineNumber":738,"sourceCode":"\t\tif err != nil {\n\t\t\tlog.Error(err.Error())\n\t\t\treturn vterrors.Wrapf(err, \"failed to get tablet connection to %s\", tabletAliasString)\n\t\t}\n\n\t\terrCh := make(chan error, 1)\n\t\tgo func() {\n\t\t\t_ = tabletConn.StreamHealth(ctx, func(shr *querypb.StreamHealthResponse) error {\n\t\t\t\tvar err error\n\t\t\t\tswitch {\n\t\t\t\tcase ctx.Err() != nil:\n\t\t\t\t\terr = vterrors.Wrapf(ctx.Err(), \"context ended while streaming tablet health from %s\", tabletAliasString)\n\t\t\t\tcase shr == nil || shr.RealtimeStats == nil || shr.Target == nil:\n\t\t\t\t\terr = fmt.Errorf(\"health check failed on %s\", tabletAliasString)\n\t\t\t\tcase vs.tabletType != shr.Target.TabletType:\n\t\t\t\t\terr = fmt.Errorf(\"tablet %s type has changed from %s to %s, restarting vstream\",\n\t\t\t\t\t\ttopoproto.TabletAliasString(tablet.Alias), vs.tabletType, shr.Target.TabletType)\n\t\t\t\tcase shr.RealtimeStats.HealthError != \"\":\n\t\t\t\t\terr = fmt.Errorf(\"tablet %s is no longer healthy: %s, restarting vstream\",\n\t\t\t\t\t\ttopoproto.TabletAliasString(tablet.Alias), shr.RealtimeStats.HealthError)\n\t\t\t\tcase shr.RealtimeStats.ReplicationLagSeconds > uint32(discovery.GetLowReplicationLag().Seconds()):\n\t\t\t\t\terr = fmt.Errorf(\"tablet %s has a replication lag of %d seconds which is beyond the value provided in --discovery_low_replication_lag of %s so the tablet is no longer considered healthy, restarting vstream\",\n\t\t\t\t\t\ttopoproto.TabletAliasString(tablet.Alias), shr.RealtimeStats.ReplicationLagSeconds, discovery.GetLowReplicationLag())\n\t\t\t\t}\n\t\t\t\tif err != nil {\n\t\t\t\t\tlog.Warn(fmt.Sprintf(\"Tablet state changed: %s, attempting to restart\", err))\n\t\t\t\t\terr = vterrors.Wrapf(err, \"error streaming tablet health from %s\", tabletAliasString)\n\t\t\t\t\terrCh <- err\n\t\t\t\t\treturn err\n\t\t\t\t}\n\t\t\t\treturn nil\n\t\t\t})\n\t\t}()\n\n\t\tvar options *binlogdatapb.VStreamOptions\n\t\tconst SidecarDBHeartbeatTableName = \"heartbeat\"\n\t\tif vs.flags.GetStreamKeyspaceHeartbeats() {","sourceCodeStart":720,"sourceCodeEnd":756,"githubUrl":"https://github.com/vitessio/vitess/blob/01a25a7d176f94613b8d59d799f438380a8760e4/go/vt/vtgate/vstream_manager.go#L720-L756","documentation":"The tablet's health stream reports RealtimeStats.HealthError (the tablet itself says it is unhealthy, e.g. MySQL down, replication broken), so the VStreamer errors 'tablet %s is no longer healthy: %s, restarting vstream'. The inner message comes from the tablet's health check (mysql ping, replication status); VTGate wraps it and restarts streaming elsewhere.","triggerScenarios":"RealtimeStats.HealthError becomes non-empty on the streaming tablet — MySQL restart, disk full, replication stopped, semi-sync timeout — during any VReplication workflow.","commonSituations":"Source MySQL crash or OOM-kill mid-copy; replica lagging/broken replication; tablet marked not serving after maintenance; storage failures on the source host.","solutions":["Read the inner HealthError text — fix the underlying tablet/MySQL issue (restart mysqld, fix replication, free disk).","Confirm tablet health endpoint shows healthy (localhost:15100/healthz).","Check replication status on the tablet (SHOW REPLICA STATUS) and repair if broken.","Let the vstream restart on a healthy tablet; verify workflow progress resumes (VReplication workflow status).","If one tablet repeatedly reports HealthError, investigate hardware/OS or take it out of the serving pool."],"exampleFix":"// before\nERROR 2002 mysqld down → HealthError=\"dial tcp ... connection refused\"\n// after\nsystemctl restart mysqld; vtctldclient SetWritable alias true; workflow auto-restarts","handlingStrategy":"retry","validationCode":"// pre-check tablet health before streaming:\n// curl http://<tablet>:15100/healthz → OK\n// SHOW REPLICA STATUS on source → no Last_Error","typeGuard":null,"tryCatchPattern":"err := streamHealth(ctx)\nif err != nil && strings.Contains(err.Error(), \"no longer healthy\") {\n    // inner HealthError tells the root cause; fix then retry\n    log.Warn(\"tablet unhealthy\", slog.Any(\"error\", err))\n    return retryWithBackoff(ctx)\n}","preventionTips":["Alert on RealtimeStats.HealthError per tablet","Monitor MySQL uptime/disk/replication on source hosts","Run VReplication workflows only on healthy clusters","Use semi-sync/HA settings that fail fast"],"tags":["vreplication","vstream","health-check","mysql"],"backgroundTag":"tablet-health-check-failed","analyzedSha":"01a25a7d176f94613b8d59d799f438380a8760e4","analyzedAt":"2026-09-01T17:28:30.605Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}