vitessio/vitess · error

context was done before filtered replication did catch up. L

Error message

context was done before filtered replication did catch up. Last seen delay: %v context Error: %v

What it means

After the StreamHealth call returns without error, WaitForFilteredReplication checks the context; if the context was cancelled or timed out, replication never caught up within the allotted time and the wait fails with the last observed delay and the context error. This is the timeout path of the catch-up wait.

Source

Thrown at go/vt/wrangler/split.go:140

		delaySecs := stats.FilteredReplicationLagSeconds
		lastSeenDelay = time.Duration(delaySecs) * time.Second
		if lastSeenDelay < 0 {
			return fmt.Errorf("last seen delay should never be negative. tablet: %v delay: %v", alias, lastSeenDelay)
		}
		if lastSeenDelay <= maxDelay {
			wr.Logger().Printf("Filtered replication on tablet: %v has caught up. Last seen delay: %.1f seconds\n", alias, lastSeenDelay.Seconds())
			return io.EOF
		}
		wr.Logger().Printf("Waiting for filtered replication to catch up on tablet: %v Last seen delay: %.1f seconds\n", alias, lastSeenDelay.Seconds())
		return nil
	})
	if err != nil {
		return fmt.Errorf("could not stream health records from tablet: %v err: %v", alias, err)
	}

	select {
	case <-ctx.Done():
		return fmt.Errorf("context was done before filtered replication did catch up. Last seen delay: %v context Error: %v", lastSeenDelay, ctx.Err())
	default:
	}
	return nil
}

View on GitHub (pinned to 01a25a7d17)

Solutions

  1. Re-run with a longer context deadline so replication has time to catch up
  2. Investigate why replication lag is high (check FilteredReplicationLagSeconds trend, vreplication throughput)
  3. Reduce the backlog (increase catch-up time, throttle writes on source, or wait for bulk load to finish)
  4. Cancel and recreate the workflow if the backlog is unbounded

Example fix

// before: short deadline causes premature timeout
ctx, cancel := context.WithTimeout(ctx, 30*time.Second)
// after: allow enough time for catch-up
ctx, cancel := context.WithTimeout(ctx, 30*time.Minute)
err := wr.WaitForFilteredReplication(ctx, alias, maxDelay)
Defensive patterns

Strategy: retry

Validate before calling

lag := currentFilteredLag(ctx, wr, alias)
if lag > maxDelay && timeUntilDeadline(ctx) < estimateCatchUp(lag) {
	return fmt.Errorf("lag %d needs ~%s; extend context deadline", lag, estimateCatchUp(lag))
}

Try / catch

if err := wr.WaitForFilteredReplication(ctx, alias, maxDelay); err != nil {
	if strings.Contains(err.Error(), "context was done before filtered replication") {
		// extend deadline and resume waiting
		ctx, cancel = context.WithTimeout(context.WithoutCancel(ctx), 30*time.Minute)
		defer cancel()
		return wr.WaitForFilteredReplication(ctx, alias, maxDelay)
	}
	return err
}

Prevention

When it happens

Trigger: Calling WaitForFilteredReplication with a context whose deadline expires (or is cancelled) before FilteredReplicationLagSeconds drops to <= maxDelay.

Common situations: Filtered replication lagging badly (large backlogs, bulk loads) during a migration while the operator's context deadline is too short; vtaction/vtctl default timeouts expiring on long catch-ups; upstream cancellation of a long-running migration command.

Related errors


AI-assisted analysis of vitessio/vitess@01a25a7d17 (2026-09-01). Data as JSON: /api/errors/9c9a178b23e2f30f. Report an issue: GitHub.