{"record":{"id":"9c9a178b23e2f30f","repo":"vitessio/vitess","slug":"context-was-done-before-filtered-replication-did-c","errorCode":null,"errorMessage":"context was done before filtered replication did catch up. Last seen delay: %v context Error: %v","messagePattern":"context was done before filtered replication did catch up\\. Last seen delay: (.+?) context Error: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"go/vt/wrangler/split.go","lineNumber":140,"sourceCode":"\t\tdelaySecs := stats.FilteredReplicationLagSeconds\n\t\tlastSeenDelay = time.Duration(delaySecs) * time.Second\n\t\tif lastSeenDelay < 0 {\n\t\t\treturn fmt.Errorf(\"last seen delay should never be negative. tablet: %v delay: %v\", alias, lastSeenDelay)\n\t\t}\n\t\tif lastSeenDelay <= maxDelay {\n\t\t\twr.Logger().Printf(\"Filtered replication on tablet: %v has caught up. Last seen delay: %.1f seconds\\n\", alias, lastSeenDelay.Seconds())\n\t\t\treturn io.EOF\n\t\t}\n\t\twr.Logger().Printf(\"Waiting for filtered replication to catch up on tablet: %v Last seen delay: %.1f seconds\\n\", alias, lastSeenDelay.Seconds())\n\t\treturn nil\n\t})\n\tif err != nil {\n\t\treturn fmt.Errorf(\"could not stream health records from tablet: %v err: %v\", alias, err)\n\t}\n\n\tselect {\n\tcase <-ctx.Done():\n\t\treturn fmt.Errorf(\"context was done before filtered replication did catch up. Last seen delay: %v context Error: %v\", lastSeenDelay, ctx.Err())\n\tdefault:\n\t}\n\treturn nil\n}\n","sourceCodeStart":122,"sourceCodeEnd":145,"githubUrl":"https://github.com/vitessio/vitess/blob/01a25a7d176f94613b8d59d799f438380a8760e4/go/vt/wrangler/split.go#L122-L145","documentation":"After the StreamHealth call returns without error, WaitForFilteredReplication checks the context; if the context was cancelled or timed out, replication never caught up within the allotted time and the wait fails with the last observed delay and the context error. This is the timeout path of the catch-up wait.","triggerScenarios":"Calling WaitForFilteredReplication with a context whose deadline expires (or is cancelled) before FilteredReplicationLagSeconds drops to <= maxDelay.","commonSituations":"Filtered replication lagging badly (large backlogs, bulk loads) during a migration while the operator's context deadline is too short; vtaction/vtctl default timeouts expiring on long catch-ups; upstream cancellation of a long-running migration command.","solutions":["Re-run with a longer context deadline so replication has time to catch up","Investigate why replication lag is high (check FilteredReplicationLagSeconds trend, vreplication throughput)","Reduce the backlog (increase catch-up time, throttle writes on source, or wait for bulk load to finish)","Cancel and recreate the workflow if the backlog is unbounded"],"exampleFix":"// before: short deadline causes premature timeout\nctx, cancel := context.WithTimeout(ctx, 30*time.Second)\n// after: allow enough time for catch-up\nctx, cancel := context.WithTimeout(ctx, 30*time.Minute)\nerr := wr.WaitForFilteredReplication(ctx, alias, maxDelay)","handlingStrategy":"retry","validationCode":"lag := currentFilteredLag(ctx, wr, alias)\nif lag > maxDelay && timeUntilDeadline(ctx) < estimateCatchUp(lag) {\n\treturn fmt.Errorf(\"lag %d needs ~%s; extend context deadline\", lag, estimateCatchUp(lag))\n}","typeGuard":null,"tryCatchPattern":"if err := wr.WaitForFilteredReplication(ctx, alias, maxDelay); err != nil {\n\tif strings.Contains(err.Error(), \"context was done before filtered replication\") {\n\t\t// extend deadline and resume waiting\n\t\tctx, cancel = context.WithTimeout(context.WithoutCancel(ctx), 30*time.Minute)\n\t\tdefer cancel()\n\t\treturn wr.WaitForFilteredReplication(ctx, alias, maxDelay)\n\t}\n\treturn err\n}","preventionTips":["Size context deadlines to expected replication backlogs","Track FilteredReplicationLagSeconds trend before switching traffic","Throttle source writes during catch-up windows"],"tags":["timeout","replication","context"],"backgroundTag":"replication-catchup-timeout","analyzedSha":"01a25a7d176f94613b8d59d799f438380a8760e4","analyzedAt":"2026-09-01T17:28:30.605Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}