What causes "could not stream health records from tablet: %v err: %v" in vitessio/vitess?

Inspect the wrapped err to see the root cause (connection failure vs callback error)

could not stream health records from tablet: %v err: %v

Error message

could not stream health records from tablet: %v err: %v

What it means

WaitForFilteredReplication wraps any error returned by the StreamHealth call itself — connection failure, stream terminated by a callback error, RPC errors — into this message. It means the health streaming session with the tablet ended abnormally before replication caught up. The inner err usually carries the real cause (often one of the callback errors above).

Source

Thrown at go/vt/wrangler/split.go:135

		}
		if stats.BinlogPlayersCount == 0 {
			return fmt.Errorf("no filtered replication running on tablet: %v health record: %v", alias, shr)
		}

		delaySecs := stats.FilteredReplicationLagSeconds
		lastSeenDelay = time.Duration(delaySecs) * time.Second
		if lastSeenDelay < 0 {
			return fmt.Errorf("last seen delay should never be negative. tablet: %v delay: %v", alias, lastSeenDelay)
		}
		if lastSeenDelay <= maxDelay {
			wr.Logger().Printf("Filtered replication on tablet: %v has caught up. Last seen delay: %.1f seconds\n", alias, lastSeenDelay.Seconds())
			return io.EOF
		}
		wr.Logger().Printf("Waiting for filtered replication to catch up on tablet: %v Last seen delay: %.1f seconds\n", alias, lastSeenDelay.Seconds())
		return nil
	})
	if err != nil {
		return fmt.Errorf("could not stream health records from tablet: %v err: %v", alias, err)
	}

	select {
	case <-ctx.Done():
		return fmt.Errorf("context was done before filtered replication did catch up. Last seen delay: %v context Error: %v", lastSeenDelay, ctx.Err())
	default:
	}
	return nil
}

View on GitHub (pinned to 01a25a7d17)

Solutions

Inspect the wrapped err to see the root cause (connection failure vs callback error)
Verify network connectivity and that the tablet's gRPC port is open
Confirm the tablet is running and registered in topology, then retry WaitForFilteredReplication
If caused by a callback error (e.g. no binlog players), fix the underlying workflow state first

Example fix

// before: opaque wrapped error
err := wr.WaitForFilteredReplication(ctx, alias, maxDelay)
// after: check tablet reachability first
if err := pingTablet(ctx, wr, alias); err != nil {
	return fmt.Errorf("tablet %s unreachable: %w", alias, err)
}
err = wr.WaitForFilteredReplication(ctx, alias, maxDelay)

Defensive patterns

Strategy: retry

Validate before calling

if err := pingTabletGRPC(ctx, alias); err != nil {
	return fmt.Errorf("tablet %s gRPC unreachable: %w", alias, err)
}

Try / catch

err := wr.WaitForFilteredReplication(ctx, alias, maxDelay)
for retries := 0; err != nil && strings.Contains(err.Error(), "could not stream health records") && retries < 3; retries++ {
	time.Sleep(2 * time.Second)
	err = wr.WaitForFilteredReplication(ctx, alias, maxDelay)
}
return err

Prevention

Verify tablet gRPC ports and network paths before migrations
Avoid restarting tablets mid-wait
Inspect the wrapped inner error to distinguish stream drops from callback failures

When it happens

Trigger: StreamHealth RPC fails to connect to the tablet, the connection drops mid-stream, or the callback returned an error (unhealthy tablet, no binlog players, etc.) that terminates the stream.

Common situations: Tablet restarted or network partition during a MoveTables migration; vtctld cannot reach the tablet's gRPC port; the callback errors (health error, no players) bubbling up through the stream.

Related errors

AI-assisted analysis of vitessio/vitess@01a25a7d17 (2026-09-01). Data as JSON: /api/errors/324638d1168d7ce4. Report an issue: GitHub.