{"record":{"id":"07a147873747cf37","repo":"vitessio/vitess","slug":"tablet-is-not-healthy-tablet-v-health-record","errorCode":null,"errorMessage":"tablet is not healthy. tablet: %v health record: %v","messagePattern":"tablet is not healthy\\. tablet: (.+?) health record: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"go/vt/wrangler/split.go","lineNumber":116,"sourceCode":"\t// This is especially true for tests and automation where there is no pause of multiple seconds\n\t// between commands and the periodic healthcheck did not run again yet.\n\tif err := wr.TabletManagerClient().RunHealthCheck(ctx, tabletInfo.Tablet); err != nil {\n\t\treturn fmt.Errorf(\"failed to run explicit healthcheck on tablet: %v err: %v\", tabletInfo, err)\n\t}\n\n\tconn, err := tabletconn.GetDialer()(ctx, tabletInfo.Tablet, grpcclient.FailFast(false))\n\tif err != nil {\n\t\treturn fmt.Errorf(\"cannot connect to tablet %v: %v\", alias, err)\n\t}\n\n\tvar lastSeenDelay time.Duration\n\terr = conn.StreamHealth(ctx, func(shr *querypb.StreamHealthResponse) error {\n\t\tstats := shr.RealtimeStats\n\t\tif stats == nil {\n\t\t\treturn fmt.Errorf(\"health record does not include RealtimeStats message. tablet: %v health record: %v\", alias, shr)\n\t\t}\n\t\tif stats.HealthError != \"\" {\n\t\t\treturn fmt.Errorf(\"tablet is not healthy. tablet: %v health record: %v\", alias, shr)\n\t\t}\n\t\tif stats.BinlogPlayersCount == 0 {\n\t\t\treturn fmt.Errorf(\"no filtered replication running on tablet: %v health record: %v\", alias, shr)\n\t\t}\n\n\t\tdelaySecs := stats.FilteredReplicationLagSeconds\n\t\tlastSeenDelay = time.Duration(delaySecs) * time.Second\n\t\tif lastSeenDelay < 0 {\n\t\t\treturn fmt.Errorf(\"last seen delay should never be negative. tablet: %v delay: %v\", alias, lastSeenDelay)\n\t\t}\n\t\tif lastSeenDelay <= maxDelay {\n\t\t\twr.Logger().Printf(\"Filtered replication on tablet: %v has caught up. Last seen delay: %.1f seconds\\n\", alias, lastSeenDelay.Seconds())\n\t\t\treturn io.EOF\n\t\t}\n\t\twr.Logger().Printf(\"Waiting for filtered replication to catch up on tablet: %v Last seen delay: %.1f seconds\\n\", alias, lastSeenDelay.Seconds())\n\t\treturn nil\n\t})\n\tif err != nil {","sourceCodeStart":98,"sourceCodeEnd":134,"githubUrl":"https://github.com/vitessio/vitess/blob/01a25a7d176f94613b8d59d799f438380a8760e4/go/vt/wrangler/split.go#L98-L134","documentation":"While waiting for filtered (vreplication) replication to catch up, the streamed health record reports a non-empty HealthError, meaning the tablet itself considers itself unhealthy. The wait is aborted immediately because an unhealthy tablet's replication lag is meaningless. The error includes the full health record for diagnosis.","triggerScenarios":"Calling WaitForFilteredReplication on a tablet whose RealtimeStats.HealthError is set — the tablet's underlying MySQL replica/health check is failing (e.g. replication stopped, MySQL down).","commonSituations":"Filtered replication lag monitoring during a MoveTables migration when the tablet's MySQL replica thread has stopped; disk full or MySQL restarted on the target tablet; health check failing due to MySQL auth or replication errors.","solutions":["Read the HealthError inside the printed health record to find the underlying MySQL/tablet problem","Fix the tablet health issue (restart replication with `vtctldclient ReplicationStart`, fix MySQL, free disk)","Re-run WaitForFilteredReplication once the tablet reports healthy","Check `vtctldclient GetTablet` and tablet /debug/vars for health status before retrying"],"exampleFix":"// before: waiting on an unhealthy tablet fails immediately\nerr := wr.WaitForFilteredReplication(ctx, alias, maxDelay)\n// after: assert health first so the failure is caught at the right layer\nhc, err := wr.VREngine().TabletHealth(ctx, alias)\nif err != nil || hc.HealthError != \"\" {\n\treturn fmt.Errorf(\"fix tablet health before waiting: %w\", err)\n}\nerr = wr.WaitForFilteredReplication(ctx, alias, maxDelay)","handlingStrategy":"validation","validationCode":"shr, err := getLatestHealth(ctx, wr, alias)\nif err != nil || shr.GetRealtimeStats().HealthError != \"\" {\n\treturn fmt.Errorf(\"tablet %s unhealthy: %v\", alias, shr.GetRealtimeStats().GetHealthError())\n}","typeGuard":"func tabletHealthy(shr *querypb.StreamHealthResponse) bool {\n\treturn shr.GetRealtimeStats() != nil && shr.GetRealtimeStats().HealthError == \"\"\n}","tryCatchPattern":"if err := wr.WaitForFilteredReplication(ctx, alias, maxDelay); err != nil {\n\tif strings.Contains(err.Error(), \"tablet is not healthy\") {\n\t\t// parse health record, fix MySQL/replication, retry\n\t}\n\treturn err\n}","preventionTips":["Monitor tablet health errors with alerting before migrations","Ensure MySQL replication is running on target tablets","Run health checks as a pre-flight step in migration tooling"],"tags":["health-check","replication","mysql"],"backgroundTag":"tablet-unhealthy","analyzedSha":"01a25a7d176f94613b8d59d799f438380a8760e4","analyzedAt":"2026-09-01T17:28:30.605Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}