{"record":{"id":"324638d1168d7ce4","repo":"vitessio/vitess","slug":"could-not-stream-health-records-from-tablet-v-er","errorCode":null,"errorMessage":"could not stream health records from tablet: %v err: %v","messagePattern":"could not stream health records from tablet: (.+?) err: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"go/vt/wrangler/split.go","lineNumber":135,"sourceCode":"\t\t}\n\t\tif stats.BinlogPlayersCount == 0 {\n\t\t\treturn fmt.Errorf(\"no filtered replication running on tablet: %v health record: %v\", alias, shr)\n\t\t}\n\n\t\tdelaySecs := stats.FilteredReplicationLagSeconds\n\t\tlastSeenDelay = time.Duration(delaySecs) * time.Second\n\t\tif lastSeenDelay < 0 {\n\t\t\treturn fmt.Errorf(\"last seen delay should never be negative. tablet: %v delay: %v\", alias, lastSeenDelay)\n\t\t}\n\t\tif lastSeenDelay <= maxDelay {\n\t\t\twr.Logger().Printf(\"Filtered replication on tablet: %v has caught up. Last seen delay: %.1f seconds\\n\", alias, lastSeenDelay.Seconds())\n\t\t\treturn io.EOF\n\t\t}\n\t\twr.Logger().Printf(\"Waiting for filtered replication to catch up on tablet: %v Last seen delay: %.1f seconds\\n\", alias, lastSeenDelay.Seconds())\n\t\treturn nil\n\t})\n\tif err != nil {\n\t\treturn fmt.Errorf(\"could not stream health records from tablet: %v err: %v\", alias, err)\n\t}\n\n\tselect {\n\tcase <-ctx.Done():\n\t\treturn fmt.Errorf(\"context was done before filtered replication did catch up. Last seen delay: %v context Error: %v\", lastSeenDelay, ctx.Err())\n\tdefault:\n\t}\n\treturn nil\n}\n","sourceCodeStart":117,"sourceCodeEnd":145,"githubUrl":"https://github.com/vitessio/vitess/blob/01a25a7d176f94613b8d59d799f438380a8760e4/go/vt/wrangler/split.go#L117-L145","documentation":"WaitForFilteredReplication wraps any error returned by the StreamHealth call itself — connection failure, stream terminated by a callback error, RPC errors — into this message. It means the health streaming session with the tablet ended abnormally before replication caught up. The inner err usually carries the real cause (often one of the callback errors above).","triggerScenarios":"StreamHealth RPC fails to connect to the tablet, the connection drops mid-stream, or the callback returned an error (unhealthy tablet, no binlog players, etc.) that terminates the stream.","commonSituations":"Tablet restarted or network partition during a MoveTables migration; vtctld cannot reach the tablet's gRPC port; the callback errors (health error, no players) bubbling up through the stream.","solutions":["Inspect the wrapped err to see the root cause (connection failure vs callback error)","Verify network connectivity and that the tablet's gRPC port is open","Confirm the tablet is running and registered in topology, then retry WaitForFilteredReplication","If caused by a callback error (e.g. no binlog players), fix the underlying workflow state first"],"exampleFix":"// before: opaque wrapped error\nerr := wr.WaitForFilteredReplication(ctx, alias, maxDelay)\n// after: check tablet reachability first\nif err := pingTablet(ctx, wr, alias); err != nil {\n\treturn fmt.Errorf(\"tablet %s unreachable: %w\", alias, err)\n}\nerr = wr.WaitForFilteredReplication(ctx, alias, maxDelay)","handlingStrategy":"retry","validationCode":"if err := pingTabletGRPC(ctx, alias); err != nil {\n\treturn fmt.Errorf(\"tablet %s gRPC unreachable: %w\", alias, err)\n}","typeGuard":null,"tryCatchPattern":"err := wr.WaitForFilteredReplication(ctx, alias, maxDelay)\nfor retries := 0; err != nil && strings.Contains(err.Error(), \"could not stream health records\") && retries < 3; retries++ {\n\ttime.Sleep(2 * time.Second)\n\terr = wr.WaitForFilteredReplication(ctx, alias, maxDelay)\n}\nreturn err","preventionTips":["Verify tablet gRPC ports and network paths before migrations","Avoid restarting tablets mid-wait","Inspect the wrapped inner error to distinguish stream drops from callback failures"],"tags":["network","streaming","health-check"],"backgroundTag":"health-stream-failed","analyzedSha":"01a25a7d176f94613b8d59d799f438380a8760e4","analyzedAt":"2026-09-01T17:28:30.605Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}