{"record":{"id":"cfadc4fa850f5beb","repo":"vitessio/vitess","slug":"aborting-s-primary-mitigation-is-required","errorCode":null,"errorMessage":"aborting %s, primary mitigation is required","messagePattern":"aborting (.+?), primary mitigation is required","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"go/vt/vtorc/logic/topology_recovery.go","lineNumber":1245,"sourceCode":"\n\t// checking if the original analysis is valid even after the primary refresh.\n\talreadyFixed, _, err := checkIfAlreadyFixed(analysisEntry)\n\tif err != nil {\n\t\tlog.Info(fmt.Sprintf(\"recheckPrimaryHealth: Checking if recovery is required returned err: %v\", err))\n\t\treturn err\n\t}\n\n\tif !alreadyFixed {\n\t\treturn nil\n\t}\n\n\t// The original analysis for the tablet has changed.\n\t// This could mean that either the original analysis has changed or some other\n\t// VTOrc instance has already performing the mitigation.\n\t// In either case, the original analysis is stale which can be safely aborted.\n\tlog.Info(fmt.Sprintf(\"recheckPrimaryHealth: Primary recovery is required, Tablet alias: %v\", primaryTabletAlias))\n\trecoveriesSkippedCounter.Add(append(recoveryLabels, RecoverySkipPrimaryRecovery.String()), 1)\n\treturn fmt.Errorf(\"aborting %s, primary mitigation is required\", originalAnalysisEntry)\n}\n\n// checkIfAlreadyFixed checks whether the problem that the analysis entry represents has already been fixed by another agent or not.\n// It returns (alreadyFixed, matchedEntry, error). When the problem still exists matchedEntry is the refreshed analysis entry\n// that triggered the same recovery; callers can use it to re-evaluate policies (e.g. the cells-no-recovery cell gate)\n// against the post-refresh state.\n//\n// Note: GetDetectionAnalysis may suppress non-primary analyses when a shard-wide\n// action is detected. Problems that declare a dependency on the shard-wide action\n// (via BeforeAnalyses/AfterAnalyses) survive suppression and will still be found\n// here. Non-dependent problems are intentionally suppressed — the shard-wide\n// action takes priority and they will be re-detected on a future poll.\nfunc checkIfAlreadyFixed(analysisEntry *inst.DetectionAnalysis) (bool, *inst.DetectionAnalysis, error) {\n\t// Run a replication analysis again. We will check if the problem persisted\n\tanalysisEntries, err := inst.GetDetectionAnalysis(analysisEntry.AnalyzedKeyspace, analysisEntry.AnalyzedShard, &inst.DetectionAnalysisHints{})\n\tif err != nil {\n\t\treturn false, nil, err\n\t}","sourceCodeStart":1227,"sourceCodeEnd":1263,"githubUrl":"https://github.com/vitessio/vitess/blob/01a25a7d176f94613b8d59d799f438380a8760e4/go/vt/vtorc/logic/topology_recovery.go#L1227-L1263","documentation":"Before executing a primary recovery, VTOrc re-checks the health of the analyzed primary. If the refreshed analysis no longer matches (the situation changed) or another VTOrc instance already mitigated it, the original analysis is considered stale and the recovery is deliberately aborted with this error so a fresh decision is made on the next cycle.","triggerScenarios":"executeCheckAndRecoverFunction re-evaluates an analysis entry via recheckPrimaryHealth and finds the primary now requires its own mitigation (or the analysis changed), so the pending recovery for the original entry is abandoned.","commonSituations":"Running multiple VTOrc instances against the same cluster where one finished the repair first; the recovered tablet recovered on its own between detection and execution; stale analysis from before a failover completed.","solutions":["No action needed — this is a deliberate safety abort; VTOrc will re-analyze and start a new recovery if still required","Check the VTOrc/prometheus counter recoveries_skipped (RecoverySkipPrimaryRecovery) to confirm it was a benign skip","If it fires constantly, ensure only one VTOrc instance watches each keyspace or reduce recovery concurrency between instances","Investigate why the primary keeps needing mitigation if the abort repeats every cycle"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"if err := recover(...); err != nil && strings.Contains(err.Error(), \"primary mitigation is required\") {\n    // benign abort: wait for next analysis cycle\n    return nil\n}","preventionTips":["Run a single VTOrc instance per cluster or coordinate recovery ownership","Monitor the recoveries_skipped metric","Keep analysis intervals consistent across VTOrc instances"],"tags":["vtorc","topology-recovery","stale-analysis"],"backgroundTag":"stale-recovery-analysis","analyzedSha":"01a25a7d176f94613b8d59d799f438380a8760e4","analyzedAt":"2026-09-01T17:28:30.605Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}