{"record":{"id":"a7ffc807abe95110","repo":"vitessio/vitess","slug":"health-record-does-not-include-realtimestats-messa","errorCode":null,"errorMessage":"health record does not include RealtimeStats message. tablet: %v health record: %v","messagePattern":"health record does not include RealtimeStats message\\. tablet: (.+?) health record: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"go/vt/wrangler/split.go","lineNumber":113,"sourceCode":"\t}\n\n\t// Always run an explicit healthcheck first to make sure we don't see any outdated values.\n\t// This is especially true for tests and automation where there is no pause of multiple seconds\n\t// between commands and the periodic healthcheck did not run again yet.\n\tif err := wr.TabletManagerClient().RunHealthCheck(ctx, tabletInfo.Tablet); err != nil {\n\t\treturn fmt.Errorf(\"failed to run explicit healthcheck on tablet: %v err: %v\", tabletInfo, err)\n\t}\n\n\tconn, err := tabletconn.GetDialer()(ctx, tabletInfo.Tablet, grpcclient.FailFast(false))\n\tif err != nil {\n\t\treturn fmt.Errorf(\"cannot connect to tablet %v: %v\", alias, err)\n\t}\n\n\tvar lastSeenDelay time.Duration\n\terr = conn.StreamHealth(ctx, func(shr *querypb.StreamHealthResponse) error {\n\t\tstats := shr.RealtimeStats\n\t\tif stats == nil {\n\t\t\treturn fmt.Errorf(\"health record does not include RealtimeStats message. tablet: %v health record: %v\", alias, shr)\n\t\t}\n\t\tif stats.HealthError != \"\" {\n\t\t\treturn fmt.Errorf(\"tablet is not healthy. tablet: %v health record: %v\", alias, shr)\n\t\t}\n\t\tif stats.BinlogPlayersCount == 0 {\n\t\t\treturn fmt.Errorf(\"no filtered replication running on tablet: %v health record: %v\", alias, shr)\n\t\t}\n\n\t\tdelaySecs := stats.FilteredReplicationLagSeconds\n\t\tlastSeenDelay = time.Duration(delaySecs) * time.Second\n\t\tif lastSeenDelay < 0 {\n\t\t\treturn fmt.Errorf(\"last seen delay should never be negative. tablet: %v delay: %v\", alias, lastSeenDelay)\n\t\t}\n\t\tif lastSeenDelay <= maxDelay {\n\t\t\twr.Logger().Printf(\"Filtered replication on tablet: %v has caught up. Last seen delay: %.1f seconds\\n\", alias, lastSeenDelay.Seconds())\n\t\t\treturn io.EOF\n\t\t}\n\t\twr.Logger().Printf(\"Waiting for filtered replication to catch up on tablet: %v Last seen delay: %.1f seconds\\n\", alias, lastSeenDelay.Seconds())","sourceCodeStart":95,"sourceCodeEnd":131,"githubUrl":"https://github.com/vitessio/vitess/blob/01a25a7d176f94613b8d59d799f438380a8760e4/go/vt/wrangler/split.go#L95-L131","documentation":"During WaitForFilteredReplication, the wrangler streams health records from a tablet via StreamHealth and requires each response to carry a RealtimeStats message. If a StreamHealthResponse arrives with RealtimeStats == nil, there is no health data to evaluate, so the callback aborts with this error. This indicates the tablet is sending health streams but omitting the stats payload, typically a tablet-side anomaly or a very early/terminal stream message.","triggerScenarios":"Calling wrangler.WaitForFilteredReplication (via vtctlcommand WaitForFilteredReplication / commandWaitForFilteredReplication) against a tablet whose StreamHealthResponse frames lack RealtimeStats, e.g. a tablet running an old binary or one whose health stream closes/misses the stats message.","commonSituations":"Mixed-version clusters where a source tablet streams health without RealtimeStats populated; tablet health stream terminating immediately after connect; vttablet misconfiguration disabling realtime stats reporting.","solutions":["Check the tablet's version and restart vttablet so it emits RealtimeStats in StreamHealth responses","Verify with `vtctldclient GetTablet <alias>` / health check that the tablet is serving and healthy","Retry WaitForFilteredReplication after confirming the tablet's health stream works (vtctl HealthCheck or tablet debug vars)","If the tablet is being decommissioned mid-migration, wait for it to stabilize or exclude it from the workflow"],"exampleFix":"// before: blindly waiting for filtered replication on a possibly misbehaving tablet\nwr.WaitForFilteredReplication(ctx, alias, maxDelay)\n// after: pre-check tablet health before streaming\nif err := checkTabletHealth(ctx, wr, alias); err != nil {\n\treturn fmt.Errorf(\"tablet %s unhealthy before wait: %w\", alias, err)\n}\nwr.WaitForFilteredReplication(ctx, alias, maxDelay)","handlingStrategy":"validation","validationCode":"hc, err := getTabletHealth(ctx, wr, alias)\nif err != nil || hc.RealtimeStats == nil {\n\treturn fmt.Errorf(\"tablet %s does not emit RealtimeStats; fix vttablet before waiting\", alias)\n}","typeGuard":"func hasRealtimeStats(shr *querypb.StreamHealthResponse) bool {\n\treturn shr != nil && shr.RealtimeStats != nil\n}","tryCatchPattern":"if err := wr.WaitForFilteredReplication(ctx, alias, maxDelay); err != nil {\n\tif strings.Contains(err.Error(), \"does not include RealtimeStats\") {\n\t\t// restart/upgrade vttablet, then retry\n\t}\n\treturn err\n}","preventionTips":["Keep vttablet versions current across the cluster","Verify tablet health streams work (vtctl HealthCheck) before long migration waits","Pin RealtimeStats emission in tablet config"],"tags":["health-check","replication","streaming"],"backgroundTag":"missing-realtime-stats","analyzedSha":"01a25a7d176f94613b8d59d799f438380a8760e4","analyzedAt":"2026-09-01T17:28:30.605Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}