last seen delay should never be negative. tablet: %v delay:

Error message

last seen delay should never be negative. tablet: %v delay: %v

What it means

Inside the health stream callback, the filtered replication lag (FilteredReplicationLagSeconds) is converted to a duration and sanity-checked; a negative lag is impossible under normal operation and signals corrupt/invalid stats data. The wait aborts with this defensive error rather than using the bogus value.

Source

Thrown at go/vt/wrangler/split.go:125

	}

	var lastSeenDelay time.Duration
	err = conn.StreamHealth(ctx, func(shr *querypb.StreamHealthResponse) error {
		stats := shr.RealtimeStats
		if stats == nil {
			return fmt.Errorf("health record does not include RealtimeStats message. tablet: %v health record: %v", alias, shr)
		}
		if stats.HealthError != "" {
			return fmt.Errorf("tablet is not healthy. tablet: %v health record: %v", alias, shr)
		}
		if stats.BinlogPlayersCount == 0 {
			return fmt.Errorf("no filtered replication running on tablet: %v health record: %v", alias, shr)
		}

		delaySecs := stats.FilteredReplicationLagSeconds
		lastSeenDelay = time.Duration(delaySecs) * time.Second
		if lastSeenDelay < 0 {
			return fmt.Errorf("last seen delay should never be negative. tablet: %v delay: %v", alias, lastSeenDelay)
		}
		if lastSeenDelay <= maxDelay {
			wr.Logger().Printf("Filtered replication on tablet: %v has caught up. Last seen delay: %.1f seconds\n", alias, lastSeenDelay.Seconds())
			return io.EOF
		}
		wr.Logger().Printf("Waiting for filtered replication to catch up on tablet: %v Last seen delay: %.1f seconds\n", alias, lastSeenDelay.Seconds())
		return nil
	})
	if err != nil {
		return fmt.Errorf("could not stream health records from tablet: %v err: %v", alias, err)
	}

	select {
	case <-ctx.Done():
		return fmt.Errorf("context was done before filtered replication did catch up. Last seen delay: %v context Error: %v", lastSeenDelay, ctx.Err())
	default:
	}
	return nil

View on GitHub (pinned to 01a25a7d17)

Solutions

Restart vttablet on the affected tablet to reset its stats reporting
Check the tablet's version for known RealtimeStats bugs and upgrade to a patched release
Capture the full health record from logs and file an issue if it reproduces
Retry the wait after the tablet emits a fresh, valid health record

Defensive patterns

Strategy: retry

Validate before calling

lag := latestRealtimeStats(ctx, wr, alias).FilteredReplicationLagSeconds
if lag < 0 {
	return fmt.Errorf("tablet %s reporting negative lag %d; restart vttablet", alias, lag)
}

Type guard

func validLag(stats *querypb.RealtimeStats) bool {
	return stats != nil && stats.FilteredReplicationLagSeconds >= 0
}

Try / catch

if err := wr.WaitForFilteredReplication(ctx, alias, maxDelay); err != nil {
	if strings.Contains(err.Error(), "should never be negative") {
		restartTablet(ctx, alias) // then retry once
		return wr.WaitForFilteredReplication(ctx, alias, maxDelay)
	}
	return err
}

Prevention

Restart vttablets that emit malformed stats
Upgrade to patched vitess releases
Report persistent negative-lag records as bugs with full health records

When it happens

Trigger: A StreamHealthResponse from the tablet carries a negative FilteredReplicationLagSeconds — a tablet-side stats bug or data corruption in the health record.

Common situations: Rare vttablet bug or clock/stats overflow producing negative lag; custom or patched vttablet binaries emitting malformed RealtimeStats; mixed-version clusters with incompatible health payloads.

Related errors

AI-assisted analysis of vitessio/vitess@01a25a7d17 (2026-09-01). Data as JSON: /api/errors/868c726a571a3c35. Report an issue: GitHub.