{"record":{"id":"4da0f1f1c8bcc367","repo":"vitessio/vitess","slug":"bug-replicationlagcache-did-not-return-the-lagrec","errorCode":null,"errorMessage":"BUG: replicationLagCache did not return the lagRecord for current replica: %v or a previous record of it. lastRateChange: %v replicationLagCache size: %v entries: %v","messagePattern":"BUG: replicationLagCache did not return the lagRecord for current replica: (.+?) or a previous record of it\\. lastRateChange: (.+?) replicationLagCache size: (.+?) entries: (.+?)","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"error","filePath":"go/vt/throttler/max_replication_lag_module.go","lineNumber":570,"sourceCode":"\tif minPropagationTime > minDuration {\n\t\tminDuration = minPropagationTime\n\t}\n\tif minDuration > m.config.MaxDurationBetweenIncreases() {\n\t\t// Cap the rate to a reasonable amount of time (very small increases may\n\t\t// result into a 20 minutes wait otherwise.)\n\t\tminDuration = m.config.MaxDurationBetweenIncreases()\n\t}\n\treturn minDuration\n}\n\nfunc (m *MaxReplicationLagModule) decreaseAndGuessRate(r *Result, now time.Time, lagRecordNow replicationLagRecord) {\n\t// Guess replication rate based on the difference in the replication lag of this\n\t// particular replica.\n\tlagRecordBefore := m.lagCache(lagRecordNow).atOrAfter(discovery.TabletToMapKey(lagRecordNow.Tablet), m.lastRateChange)\n\tif lagRecordBefore.isZero() {\n\t\t// We should see at least \"lagRecordNow\" here because we did just insert it\n\t\t// in processRecord().\n\t\tpanic(fmt.Sprintf(\"BUG: replicationLagCache did not return the lagRecord for current replica: %v or a previous record of it. lastRateChange: %v replicationLagCache size: %v entries: %v\", lagRecordNow, m.lastRateChange, len(m.lagCache(lagRecordNow).entries), m.lagCache(lagRecordNow).entries))\n\t}\n\t// Store the record in the result.\n\tr.LagRecordBefore = lagRecordBefore\n\tif lagRecordBefore.time.Equal(lagRecordNow.time) {\n\t\t// No lag record for this replica in the time span\n\t\t// [last rate change, current lag record).\n\t\t// Without it we won't be able to guess the replication rate.\n\t\t// We err on the side of caution and reduce the rate by half the emergency\n\t\t// decrease percentage.\n\t\tdecreaseReason := fmt.Sprintf(\"no previous lag record for this replica available since the last rate change (%.1f seconds ago)\", now.Sub(m.lastRateChange).Seconds())\n\t\tm.decreaseRateByPercentage(r, now, lagRecordNow, stateDecreaseAndGuessRate, m.config.EmergencyDecrease/2, decreaseReason)\n\t\treturn\n\t}\n\n\t// Analyze if the past rate was good or bad.\n\tlagBefore := lagRecordBefore.lag()\n\tlagNow := lagRecordNow.lag()\n\treplicationLagChange := less","sourceCodeStart":552,"sourceCodeEnd":588,"githubUrl":"https://github.com/vitessio/vitess/blob/01a25a7d176f94613b8d59d799f438380a8760e4/go/vt/throttler/max_replication_lag_module.go#L552-L588","documentation":"This panic fires in decreaseAndGuessRate when the throttler's replication-lag cache fails to return a prior lag record for the current replica, violating its internal invariant that the record just inserted by processRecord() is always retrievable. It signals cache eviction/corruption logic (atOrAfter / lastRateChange window filtering) has gone wrong, so rate recalculation cannot proceed and the process aborts deliberately.","triggerScenarios":"Calling recalculateRate on the throttler when lagCache.atOrAfter() returns a zero record for the tablet currently being processed — i.e. the just-inserted lagRecordNow for that tablet is missing from the cache or all cached entries for that tablet are older than m.lastRateChange.","commonSituations":"Concurrency bugs in the throttler's cache maintenance, a tablet key changing between insert and lookup, custom/clock-skewed time sources making lastRateChange later than every cached entry, or modifications to lagCache eviction that drop the newest record.","solutions":["Check for concurrent modification of the lagCache between processRecord() and decreaseAndGuessRate() and add proper locking","Verify lastRateChange is never set to a time at/after the just-inserted record's time","Inspect the cache eviction logic in lagCache to ensure the newest record per tablet is never dropped","Reproduce with the panic's printed entries and file an issue with the full dump"],"exampleFix":"// before (racy window)\nlagRecordBefore := m.lagCache(lagRecordNow).atOrAfter(discovery.TabletToMapKey(lagRecordNow.Tablet), m.lastRateChange)\n// after: hold the same lock across insert+lookup so the record cannot vanish\nm.mu.Lock()\nm.lagCache(lagRecordNow).addRecord(lagRecordNow)\nlagRecordBefore := m.lagCache(lagRecordNow).atOrAfter(discovery.TabletToMapKey(lagRecordNow.Tablet), m.lastRateChange)\nm.mu.Unlock()","handlingStrategy":"validation","validationCode":"if rec := m.lagCache(lagRecordNow).atOrAfter(key, m.lastRateChange); rec.isZero() {\n    // abort recalculation instead of panicking\n    return fmt.Errorf(\"no lag record for %v since %v\", key, m.lastRateChange)\n}","typeGuard":"func hasLagRecord(c lagCache, key string, since time.Time) bool {\n    return !c.atOrAfter(key, since).isZero()\n}","tryCatchPattern":null,"preventionTips":["Serialize processRecord() and recalculateRate() under the same lock","Never advance lastRateChange past the newest cached record's time","Add a unit test asserting atOrAfter finds a just-inserted record","Log the cache entries dump before aborting to aid diagnosis"],"tags":["go","panic","throttler","replication-lag","invariant-violation"],"backgroundTag":"internal-invariant-panic","analyzedSha":"01a25a7d176f94613b8d59d799f438380a8760e4","analyzedAt":"2026-09-01T17:28:30.605Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}