vitessio/vitess · error
context was done before filtered replication did catch up. L
Error message
context was done before filtered replication did catch up. Last seen delay: %v context Error: %v
What it means
After the StreamHealth call returns without error, WaitForFilteredReplication checks the context; if the context was cancelled or timed out, replication never caught up within the allotted time and the wait fails with the last observed delay and the context error. This is the timeout path of the catch-up wait.
Source
Thrown at go/vt/wrangler/split.go:140
delaySecs := stats.FilteredReplicationLagSeconds
lastSeenDelay = time.Duration(delaySecs) * time.Second
if lastSeenDelay < 0 {
return fmt.Errorf("last seen delay should never be negative. tablet: %v delay: %v", alias, lastSeenDelay)
}
if lastSeenDelay <= maxDelay {
wr.Logger().Printf("Filtered replication on tablet: %v has caught up. Last seen delay: %.1f seconds\n", alias, lastSeenDelay.Seconds())
return io.EOF
}
wr.Logger().Printf("Waiting for filtered replication to catch up on tablet: %v Last seen delay: %.1f seconds\n", alias, lastSeenDelay.Seconds())
return nil
})
if err != nil {
return fmt.Errorf("could not stream health records from tablet: %v err: %v", alias, err)
}
select {
case <-ctx.Done():
return fmt.Errorf("context was done before filtered replication did catch up. Last seen delay: %v context Error: %v", lastSeenDelay, ctx.Err())
default:
}
return nil
}
View on GitHub (pinned to 01a25a7d17)
Solutions
- Re-run with a longer context deadline so replication has time to catch up
- Investigate why replication lag is high (check FilteredReplicationLagSeconds trend, vreplication throughput)
- Reduce the backlog (increase catch-up time, throttle writes on source, or wait for bulk load to finish)
- Cancel and recreate the workflow if the backlog is unbounded
Example fix
// before: short deadline causes premature timeout ctx, cancel := context.WithTimeout(ctx, 30*time.Second) // after: allow enough time for catch-up ctx, cancel := context.WithTimeout(ctx, 30*time.Minute) err := wr.WaitForFilteredReplication(ctx, alias, maxDelay)
Defensive patterns
Strategy: retry
Validate before calling
lag := currentFilteredLag(ctx, wr, alias)
if lag > maxDelay && timeUntilDeadline(ctx) < estimateCatchUp(lag) {
return fmt.Errorf("lag %d needs ~%s; extend context deadline", lag, estimateCatchUp(lag))
} Try / catch
if err := wr.WaitForFilteredReplication(ctx, alias, maxDelay); err != nil {
if strings.Contains(err.Error(), "context was done before filtered replication") {
// extend deadline and resume waiting
ctx, cancel = context.WithTimeout(context.WithoutCancel(ctx), 30*time.Minute)
defer cancel()
return wr.WaitForFilteredReplication(ctx, alias, maxDelay)
}
return err
} Prevention
- Size context deadlines to expected replication backlogs
- Track FilteredReplicationLagSeconds trend before switching traffic
- Throttle source writes during catch-up windows
When it happens
Trigger: Calling WaitForFilteredReplication with a context whose deadline expires (or is cancelled) before FilteredReplicationLagSeconds drops to <= maxDelay.
Common situations: Filtered replication lagging badly (large backlogs, bulk loads) during a migration while the operator's context deadline is too short; vtaction/vtctl default timeouts expiring on long catch-ups; upstream cancellation of a long-running migration command.
Related errors
- clone execution failed: %v
- failed to get position after clone: %v
- %v: %v
- invalid choice for enum
- invalid key:value pair
AI-assisted analysis of vitessio/vitess@01a25a7d17 (2026-09-01).
Data as JSON: /api/errors/9c9a178b23e2f30f.
Report an issue: GitHub.