{"record":{"id":"efc96f67a7bed5db","repo":"vitessio/vitess","slug":"health-check-failed-on-s","errorCode":null,"errorMessage":"health check failed on %s","messagePattern":"health check failed on (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"go/vt/vtgate/vstream_manager.go","lineNumber":733,"sourceCode":"\t\t\tShard:      sgtid.Shard,\n\t\t\tTabletType: vs.tabletType,\n\t\t\tCell:       vs.vsm.cell,\n\t\t}\n\t\ttabletConn, err := vs.vsm.resolver.GetGateway().QueryServiceByAlias(ctx, tablet.Alias, target)\n\t\tif err != nil {\n\t\t\tlog.Error(err.Error())\n\t\t\treturn vterrors.Wrapf(err, \"failed to get tablet connection to %s\", tabletAliasString)\n\t\t}\n\n\t\terrCh := make(chan error, 1)\n\t\tgo func() {\n\t\t\t_ = tabletConn.StreamHealth(ctx, func(shr *querypb.StreamHealthResponse) error {\n\t\t\t\tvar err error\n\t\t\t\tswitch {\n\t\t\t\tcase ctx.Err() != nil:\n\t\t\t\t\terr = vterrors.Wrapf(ctx.Err(), \"context ended while streaming tablet health from %s\", tabletAliasString)\n\t\t\t\tcase shr == nil || shr.RealtimeStats == nil || shr.Target == nil:\n\t\t\t\t\terr = fmt.Errorf(\"health check failed on %s\", tabletAliasString)\n\t\t\t\tcase vs.tabletType != shr.Target.TabletType:\n\t\t\t\t\terr = fmt.Errorf(\"tablet %s type has changed from %s to %s, restarting vstream\",\n\t\t\t\t\t\ttopoproto.TabletAliasString(tablet.Alias), vs.tabletType, shr.Target.TabletType)\n\t\t\t\tcase shr.RealtimeStats.HealthError != \"\":\n\t\t\t\t\terr = fmt.Errorf(\"tablet %s is no longer healthy: %s, restarting vstream\",\n\t\t\t\t\t\ttopoproto.TabletAliasString(tablet.Alias), shr.RealtimeStats.HealthError)\n\t\t\t\tcase shr.RealtimeStats.ReplicationLagSeconds > uint32(discovery.GetLowReplicationLag().Seconds()):\n\t\t\t\t\terr = fmt.Errorf(\"tablet %s has a replication lag of %d seconds which is beyond the value provided in --discovery_low_replication_lag of %s so the tablet is no longer considered healthy, restarting vstream\",\n\t\t\t\t\t\ttopoproto.TabletAliasString(tablet.Alias), shr.RealtimeStats.ReplicationLagSeconds, discovery.GetLowReplicationLag())\n\t\t\t\t}\n\t\t\t\tif err != nil {\n\t\t\t\t\tlog.Warn(fmt.Sprintf(\"Tablet state changed: %s, attempting to restart\", err))\n\t\t\t\t\terr = vterrors.Wrapf(err, \"error streaming tablet health from %s\", tabletAliasString)\n\t\t\t\t\terrCh <- err\n\t\t\t\t\treturn err\n\t\t\t\t}\n\t\t\t\treturn nil\n\t\t\t})","sourceCodeStart":715,"sourceCodeEnd":751,"githubUrl":"https://github.com/vitessio/vitess/blob/01a25a7d176f94613b8d59d799f438380a8760e4/go/vt/vtgate/vstream_manager.go#L715-L751","documentation":"The VStreamer streams tablet health for each source tablet via StreamHealth; when a response has no RealtimeStats or Target (or shr is nil) it produces 'health check failed on %s'. This signals the tablet's health stream returned an unusable/malformed response, so the vstream logs it, wraps the error as 'error streaming tablet health from %s' and sends it to errCh, causing the vstream to restart on that tablet.","triggerScenarios":"A tablet closes/initializes the health stream without sending a well-formed StreamHealthResponse (nil response, nil RealtimeStats, or nil Target) while a VReplication workflow (MoveTables, Reshard, etc.) streams from it.","commonSituations":"Tablet restarting or being taken down during migration; network interruption truncating health stream; tablet serving a different target than expected right after reparent; version mismatch where older tablets don't populate fields.","solutions":["Check the source tablet's logs and health (vtctldclient GetTablets / tablet web UI 15100/debug vars).","Confirm the tablet is RUNNING and serving the expected tablet type; restart the tablet if wedged.","Verify tablet and vtgate/vtctld versions are compatible (upgrade older tablets).","Check network stability between vtgate and the tablet's gRPC port.","VStream will attempt to restart on a healthy tablet — if persistent, rebuild the tablet or re-init replication."],"exampleFix":"// before: tablet down during vstream\n$ vtctldclient PlannedReparentShard  # reparent failed, tablet stale\n// after: restore tablet health first\n$ vtctldclient RebuildKeyspaceGraph && restart vittablet; then retry MoveTables","handlingStrategy":"retry","validationCode":"// before starting the workflow, check tablet health:\n// vtctldclient GetTablets --keyspace ks → all sources must be SERVING\n// curl http://<tablet>:15100/healthz  → must return OK","typeGuard":null,"tryCatchPattern":"errCh := make(chan error, 1)\ngo func() { errCh <- startVStream() }()\nselect {\ncase err := <-errCh:\n    if strings.Contains(err.Error(), \"health check failed\") {\n        // wait for tablet recovery then restart the stream\n        time.Sleep(backoff)\n        retry()\n    }\n}","preventionTips":["Confirm all source tablets are healthy before MoveTables/Reshard","Keep tablet and vtgate versions in lockstep","Monitor tablet health endpoints with alerting","Avoid taking tablets down mid-workflow; pause VReplication first"],"tags":["vreplication","vstream","health-check","tablet"],"backgroundTag":"tablet-health-check-failed","analyzedSha":"01a25a7d176f94613b8d59d799f438380a8760e4","analyzedAt":"2026-09-01T17:28:30.605Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}