{"record":{"id":"868c726a571a3c35","repo":"vitessio/vitess","slug":"last-seen-delay-should-never-be-negative-tablet","errorCode":null,"errorMessage":"last seen delay should never be negative. tablet: %v delay: %v","messagePattern":"last seen delay should never be negative\\. tablet: (.+?) delay: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"go/vt/wrangler/split.go","lineNumber":125,"sourceCode":"\t}\n\n\tvar lastSeenDelay time.Duration\n\terr = conn.StreamHealth(ctx, func(shr *querypb.StreamHealthResponse) error {\n\t\tstats := shr.RealtimeStats\n\t\tif stats == nil {\n\t\t\treturn fmt.Errorf(\"health record does not include RealtimeStats message. tablet: %v health record: %v\", alias, shr)\n\t\t}\n\t\tif stats.HealthError != \"\" {\n\t\t\treturn fmt.Errorf(\"tablet is not healthy. tablet: %v health record: %v\", alias, shr)\n\t\t}\n\t\tif stats.BinlogPlayersCount == 0 {\n\t\t\treturn fmt.Errorf(\"no filtered replication running on tablet: %v health record: %v\", alias, shr)\n\t\t}\n\n\t\tdelaySecs := stats.FilteredReplicationLagSeconds\n\t\tlastSeenDelay = time.Duration(delaySecs) * time.Second\n\t\tif lastSeenDelay < 0 {\n\t\t\treturn fmt.Errorf(\"last seen delay should never be negative. tablet: %v delay: %v\", alias, lastSeenDelay)\n\t\t}\n\t\tif lastSeenDelay <= maxDelay {\n\t\t\twr.Logger().Printf(\"Filtered replication on tablet: %v has caught up. Last seen delay: %.1f seconds\\n\", alias, lastSeenDelay.Seconds())\n\t\t\treturn io.EOF\n\t\t}\n\t\twr.Logger().Printf(\"Waiting for filtered replication to catch up on tablet: %v Last seen delay: %.1f seconds\\n\", alias, lastSeenDelay.Seconds())\n\t\treturn nil\n\t})\n\tif err != nil {\n\t\treturn fmt.Errorf(\"could not stream health records from tablet: %v err: %v\", alias, err)\n\t}\n\n\tselect {\n\tcase <-ctx.Done():\n\t\treturn fmt.Errorf(\"context was done before filtered replication did catch up. Last seen delay: %v context Error: %v\", lastSeenDelay, ctx.Err())\n\tdefault:\n\t}\n\treturn nil","sourceCodeStart":107,"sourceCodeEnd":143,"githubUrl":"https://github.com/vitessio/vitess/blob/01a25a7d176f94613b8d59d799f438380a8760e4/go/vt/wrangler/split.go#L107-L143","documentation":"Inside the health stream callback, the filtered replication lag (FilteredReplicationLagSeconds) is converted to a duration and sanity-checked; a negative lag is impossible under normal operation and signals corrupt/invalid stats data. The wait aborts with this defensive error rather than using the bogus value.","triggerScenarios":"A StreamHealthResponse from the tablet carries a negative FilteredReplicationLagSeconds — a tablet-side stats bug or data corruption in the health record.","commonSituations":"Rare vttablet bug or clock/stats overflow producing negative lag; custom or patched vttablet binaries emitting malformed RealtimeStats; mixed-version clusters with incompatible health payloads.","solutions":["Restart vttablet on the affected tablet to reset its stats reporting","Check the tablet's version for known RealtimeStats bugs and upgrade to a patched release","Capture the full health record from logs and file an issue if it reproduces","Retry the wait after the tablet emits a fresh, valid health record"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"lag := latestRealtimeStats(ctx, wr, alias).FilteredReplicationLagSeconds\nif lag < 0 {\n\treturn fmt.Errorf(\"tablet %s reporting negative lag %d; restart vttablet\", alias, lag)\n}","typeGuard":"func validLag(stats *querypb.RealtimeStats) bool {\n\treturn stats != nil && stats.FilteredReplicationLagSeconds >= 0\n}","tryCatchPattern":"if err := wr.WaitForFilteredReplication(ctx, alias, maxDelay); err != nil {\n\tif strings.Contains(err.Error(), \"should never be negative\") {\n\t\trestartTablet(ctx, alias) // then retry once\n\t\treturn wr.WaitForFilteredReplication(ctx, alias, maxDelay)\n\t}\n\treturn err\n}","preventionTips":["Restart vttablets that emit malformed stats","Upgrade to patched vitess releases","Report persistent negative-lag records as bugs with full health records"],"tags":["health-check","replication","data-integrity"],"backgroundTag":"negative-replication-lag","analyzedSha":"01a25a7d176f94613b8d59d799f438380a8760e4","analyzedAt":"2026-09-01T17:28:30.605Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}