vitessio/vitess · warning

tablet %s has a replication lag of %d seconds which is beyon

Error message

tablet %s has a replication lag of %d seconds which is beyond the value provided in --discovery_low_replication_lag of %s so the tablet is no longer considered healthy, restarting vstream

What it means

When RealtimeStats.ReplicationLagSeconds exceeds the --discovery_low_replication_lag threshold, the VStreamer declares the tablet unhealthy with 'tablet %s has a replication lag of %d seconds which is beyond the value provided in --discovery_low_replication_lag of %s so the tablet is no longer considered healthy, restarting vstream'. This throttles/protects VReplication from copying from too-far-behind sources; the error is wrapped as 'error streaming tablet health from %s' and triggers a restart on another (hopefully fresher) tablet.

Source

Thrown at go/vt/vtgate/vstream_manager.go:741

		}

		errCh := make(chan error, 1)
		go func() {
			_ = tabletConn.StreamHealth(ctx, func(shr *querypb.StreamHealthResponse) error {
				var err error
				switch {
				case ctx.Err() != nil:
					err = vterrors.Wrapf(ctx.Err(), "context ended while streaming tablet health from %s", tabletAliasString)
				case shr == nil || shr.RealtimeStats == nil || shr.Target == nil:
					err = fmt.Errorf("health check failed on %s", tabletAliasString)
				case vs.tabletType != shr.Target.TabletType:
					err = fmt.Errorf("tablet %s type has changed from %s to %s, restarting vstream",
						topoproto.TabletAliasString(tablet.Alias), vs.tabletType, shr.Target.TabletType)
				case shr.RealtimeStats.HealthError != "":
					err = fmt.Errorf("tablet %s is no longer healthy: %s, restarting vstream",
						topoproto.TabletAliasString(tablet.Alias), shr.RealtimeStats.HealthError)
				case shr.RealtimeStats.ReplicationLagSeconds > uint32(discovery.GetLowReplicationLag().Seconds()):
					err = fmt.Errorf("tablet %s has a replication lag of %d seconds which is beyond the value provided in --discovery_low_replication_lag of %s so the tablet is no longer considered healthy, restarting vstream",
						topoproto.TabletAliasString(tablet.Alias), shr.RealtimeStats.ReplicationLagSeconds, discovery.GetLowReplicationLag())
				}
				if err != nil {
					log.Warn(fmt.Sprintf("Tablet state changed: %s, attempting to restart", err))
					err = vterrors.Wrapf(err, "error streaming tablet health from %s", tabletAliasString)
					errCh <- err
					return err
				}
				return nil
			})
		}()

		var options *binlogdatapb.VStreamOptions
		const SidecarDBHeartbeatTableName = "heartbeat"
		if vs.flags.GetStreamKeyspaceHeartbeats() {
			options = &binlogdatapb.VStreamOptions{
				InternalTables: []string{SidecarDBHeartbeatTableName},
			}

View on GitHub (pinned to 01a25a7d17)

Solutions

1. Tune --discovery_low_replication_lag on vtgate to a realistic threshold for your cluster.
2. Reduce write load / throttle the pipeline causing lag (pt-kill long queries, batch app writes).
3. Add capacity or fix replication throughput on the lagging tablet.
4. Verify lag with vtctldclient status / SHOW REPLICA STATUS; the vstream will retry and pick a low-lag tablet automatically.
5. If lag is acceptable for your use, raise the threshold so the stream is not restarted repeatedly.

Example fix

// before
vtgate --discovery_low_replication_lag=1s  # constant restarts on busy cluster
// after
vtgate --discovery_low_replication_lag=30s  # tolerates normal replication lag

// before starting the workflow, check lag:
// SHOW REPLICA STATUS → Seconds_Behind_Source must be < --discovery_low_replication_lag
// or vtctldclient status <alias>

if err != nil && strings.Contains(err.Error(), "replication lag") {
    // wait for lag to drop, then restart vstream
    waitUntilLowLag(ctx, threshold)
    return restartVStream(ctx)
}

When it happens

Trigger: Source replica's Seconds_Behind_Source exceeds discovery_low_replication_lag while MoveTables/Reshard/Lookup vindex backfill streams from it.

Common situations: Bulk writes saturating the replica; under-provisioned source hardware; long-running queries blocking replication; too-aggressive (low) threshold setting; replica IO thread stalls.

Related errors

• vstream ended
• stream needs a position or a table to copy
• partial row image encountered: ensure binlog_row_image is se
• failed to instantiate throttler: %v
• no binlog player client factory named %v

AI-assisted analysis of vitessio/vitess@01a25a7d17 (2026-09-01). Data as JSON: /api/errors/b65ac63cce2d2003. Report an issue: GitHub.