BUG: replicationLagCache did not return the lagRecord for cu

Error message

BUG: replicationLagCache did not return the lagRecord for current replica: %v or a previous record of it. lastRateChange: %v replicationLagCache size: %v entries: %v

What it means

This panic fires in decreaseAndGuessRate when the throttler's replication-lag cache fails to return a prior lag record for the current replica, violating its internal invariant that the record just inserted by processRecord() is always retrievable. It signals cache eviction/corruption logic (atOrAfter / lastRateChange window filtering) has gone wrong, so rate recalculation cannot proceed and the process aborts deliberately.

Source

Thrown at go/vt/throttler/max_replication_lag_module.go:570

	if minPropagationTime > minDuration {
		minDuration = minPropagationTime
	}
	if minDuration > m.config.MaxDurationBetweenIncreases() {
		// Cap the rate to a reasonable amount of time (very small increases may
		// result into a 20 minutes wait otherwise.)
		minDuration = m.config.MaxDurationBetweenIncreases()
	}
	return minDuration
}

func (m *MaxReplicationLagModule) decreaseAndGuessRate(r *Result, now time.Time, lagRecordNow replicationLagRecord) {
	// Guess replication rate based on the difference in the replication lag of this
	// particular replica.
	lagRecordBefore := m.lagCache(lagRecordNow).atOrAfter(discovery.TabletToMapKey(lagRecordNow.Tablet), m.lastRateChange)
	if lagRecordBefore.isZero() {
		// We should see at least "lagRecordNow" here because we did just insert it
		// in processRecord().
		panic(fmt.Sprintf("BUG: replicationLagCache did not return the lagRecord for current replica: %v or a previous record of it. lastRateChange: %v replicationLagCache size: %v entries: %v", lagRecordNow, m.lastRateChange, len(m.lagCache(lagRecordNow).entries), m.lagCache(lagRecordNow).entries))
	}
	// Store the record in the result.
	r.LagRecordBefore = lagRecordBefore
	if lagRecordBefore.time.Equal(lagRecordNow.time) {
		// No lag record for this replica in the time span
		// [last rate change, current lag record).
		// Without it we won't be able to guess the replication rate.
		// We err on the side of caution and reduce the rate by half the emergency
		// decrease percentage.
		decreaseReason := fmt.Sprintf("no previous lag record for this replica available since the last rate change (%.1f seconds ago)", now.Sub(m.lastRateChange).Seconds())
		m.decreaseRateByPercentage(r, now, lagRecordNow, stateDecreaseAndGuessRate, m.config.EmergencyDecrease/2, decreaseReason)
		return
	}

	// Analyze if the past rate was good or bad.
	lagBefore := lagRecordBefore.lag()
	lagNow := lagRecordNow.lag()
	replicationLagChange := less

View on GitHub (pinned to 01a25a7d17)

Solutions

Check for concurrent modification of the lagCache between processRecord() and decreaseAndGuessRate() and add proper locking
Verify lastRateChange is never set to a time at/after the just-inserted record's time
Inspect the cache eviction logic in lagCache to ensure the newest record per tablet is never dropped
Reproduce with the panic's printed entries and file an issue with the full dump

Example fix

// before (racy window)
lagRecordBefore := m.lagCache(lagRecordNow).atOrAfter(discovery.TabletToMapKey(lagRecordNow.Tablet), m.lastRateChange)
// after: hold the same lock across insert+lookup so the record cannot vanish
m.mu.Lock()
m.lagCache(lagRecordNow).addRecord(lagRecordNow)
lagRecordBefore := m.lagCache(lagRecordNow).atOrAfter(discovery.TabletToMapKey(lagRecordNow.Tablet), m.lastRateChange)
m.mu.Unlock()

Defensive patterns

Strategy: validation

Validate before calling

if rec := m.lagCache(lagRecordNow).atOrAfter(key, m.lastRateChange); rec.isZero() {
    // abort recalculation instead of panicking
    return fmt.Errorf("no lag record for %v since %v", key, m.lastRateChange)
}

Type guard

func hasLagRecord(c lagCache, key string, since time.Time) bool {
    return !c.atOrAfter(key, since).isZero()
}

Prevention

Serialize processRecord() and recalculateRate() under the same lock
Never advance lastRateChange past the newest cached record's time
Add a unit test asserting atOrAfter finds a just-inserted record
Log the cache entries dump before aborting to aid diagnosis

When it happens

Trigger: Calling recalculateRate on the throttler when lagCache.atOrAfter() returns a zero record for the tablet currently being processed — i.e. the just-inserted lagRecordNow for that tablet is missing from the cache or all cached entries for that tablet are older than m.lastRateChange.

Common situations: Concurrency bugs in the throttler's cache maintenance, a tablet key changing between insert and lookup, custom/clock-skewed time sources making lastRateChange later than every cached entry, or modifications to lagCache eviction that drop the newest record.

Related errors

AI-assisted analysis of vitessio/vitess@01a25a7d17 (2026-09-01). Data as JSON: /api/errors/4da0f1f1c8bcc367. Report an issue: GitHub.