{"record":{"id":"b65ac63cce2d2003","repo":"vitessio/vitess","slug":"tablet-s-has-a-replication-lag-of-d-seconds-whic","errorCode":null,"errorMessage":"tablet %s has a replication lag of %d seconds which is beyond the value provided in --discovery_low_replication_lag of %s so the tablet is no longer considered healthy, restarting vstream","messagePattern":"tablet (.+?) has a replication lag of (.+?) seconds which is beyond the value provided in --discovery_low_replication_lag of (.+?) so the tablet is no longer considered healthy, restarting vstream","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"go/vt/vtgate/vstream_manager.go","lineNumber":741,"sourceCode":"\t\t}\n\n\t\terrCh := make(chan error, 1)\n\t\tgo func() {\n\t\t\t_ = tabletConn.StreamHealth(ctx, func(shr *querypb.StreamHealthResponse) error {\n\t\t\t\tvar err error\n\t\t\t\tswitch {\n\t\t\t\tcase ctx.Err() != nil:\n\t\t\t\t\terr = vterrors.Wrapf(ctx.Err(), \"context ended while streaming tablet health from %s\", tabletAliasString)\n\t\t\t\tcase shr == nil || shr.RealtimeStats == nil || shr.Target == nil:\n\t\t\t\t\terr = fmt.Errorf(\"health check failed on %s\", tabletAliasString)\n\t\t\t\tcase vs.tabletType != shr.Target.TabletType:\n\t\t\t\t\terr = fmt.Errorf(\"tablet %s type has changed from %s to %s, restarting vstream\",\n\t\t\t\t\t\ttopoproto.TabletAliasString(tablet.Alias), vs.tabletType, shr.Target.TabletType)\n\t\t\t\tcase shr.RealtimeStats.HealthError != \"\":\n\t\t\t\t\terr = fmt.Errorf(\"tablet %s is no longer healthy: %s, restarting vstream\",\n\t\t\t\t\t\ttopoproto.TabletAliasString(tablet.Alias), shr.RealtimeStats.HealthError)\n\t\t\t\tcase shr.RealtimeStats.ReplicationLagSeconds > uint32(discovery.GetLowReplicationLag().Seconds()):\n\t\t\t\t\terr = fmt.Errorf(\"tablet %s has a replication lag of %d seconds which is beyond the value provided in --discovery_low_replication_lag of %s so the tablet is no longer considered healthy, restarting vstream\",\n\t\t\t\t\t\ttopoproto.TabletAliasString(tablet.Alias), shr.RealtimeStats.ReplicationLagSeconds, discovery.GetLowReplicationLag())\n\t\t\t\t}\n\t\t\t\tif err != nil {\n\t\t\t\t\tlog.Warn(fmt.Sprintf(\"Tablet state changed: %s, attempting to restart\", err))\n\t\t\t\t\terr = vterrors.Wrapf(err, \"error streaming tablet health from %s\", tabletAliasString)\n\t\t\t\t\terrCh <- err\n\t\t\t\t\treturn err\n\t\t\t\t}\n\t\t\t\treturn nil\n\t\t\t})\n\t\t}()\n\n\t\tvar options *binlogdatapb.VStreamOptions\n\t\tconst SidecarDBHeartbeatTableName = \"heartbeat\"\n\t\tif vs.flags.GetStreamKeyspaceHeartbeats() {\n\t\t\toptions = &binlogdatapb.VStreamOptions{\n\t\t\t\tInternalTables: []string{SidecarDBHeartbeatTableName},\n\t\t\t}","sourceCodeStart":723,"sourceCodeEnd":759,"githubUrl":"https://github.com/vitessio/vitess/blob/01a25a7d176f94613b8d59d799f438380a8760e4/go/vt/vtgate/vstream_manager.go#L723-L759","documentation":"When RealtimeStats.ReplicationLagSeconds exceeds the --discovery_low_replication_lag threshold, the VStreamer declares the tablet unhealthy with 'tablet %s has a replication lag of %d seconds which is beyond the value provided in --discovery_low_replication_lag of %s so the tablet is no longer considered healthy, restarting vstream'. This throttles/protects VReplication from copying from too-far-behind sources; the error is wrapped as 'error streaming tablet health from %s' and triggers a restart on another (hopefully fresher) tablet.","triggerScenarios":"Source replica's Seconds_Behind_Source exceeds discovery_low_replication_lag while MoveTables/Reshard/Lookup vindex backfill streams from it.","commonSituations":"Bulk writes saturating the replica; under-provisioned source hardware; long-running queries blocking replication; too-aggressive (low) threshold setting; replica IO thread stalls.","solutions":["Tune --discovery_low_replication_lag on vtgate to a realistic threshold for your cluster.","Reduce write load / throttle the pipeline causing lag (pt-kill long queries, batch app writes).","Add capacity or fix replication throughput on the lagging tablet.","Verify lag with vtctldclient status / SHOW REPLICA STATUS; the vstream will retry and pick a low-lag tablet automatically.","If lag is acceptable for your use, raise the threshold so the stream is not restarted repeatedly."],"exampleFix":"// before\nvtgate --discovery_low_replication_lag=1s  # constant restarts on busy cluster\n// after\nvtgate --discovery_low_replication_lag=30s  # tolerates normal replication lag","handlingStrategy":"retry","validationCode":"// before starting the workflow, check lag:\n// SHOW REPLICA STATUS → Seconds_Behind_Source must be < --discovery_low_replication_lag\n// or vtctldclient status <alias>","typeGuard":null,"tryCatchPattern":"if err != nil && strings.Contains(err.Error(), \"replication lag\") {\n    // wait for lag to drop, then restart vstream\n    waitUntilLowLag(ctx, threshold)\n    return restartVStream(ctx)\n}","preventionTips":["Size --discovery_low_replication_lag to realistic cluster lag","Throttle bulk writes during VReplication copies","Monitor Seconds_Behind_Source with alerting","Fix slow replicas before starting migrations"],"tags":["vreplication","vstream","replication-lag","configuration"],"backgroundTag":"replication-lag-exceeded","analyzedSha":"01a25a7d176f94613b8d59d799f438380a8760e4","analyzedAt":"2026-09-01T17:28:30.605Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}