{"record":{"id":"8191828e80888f72","repo":"vitessio/vitess","slug":"failed-to-read-instance-w","errorCode":null,"errorMessage":"failed to read instance: %w","messagePattern":"failed to read instance: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"go/vt/vtorc/logic/topology_recovery.go","lineNumber":1576,"sourceCode":"\t\tslog.String(\"tablet\", aliasString),\n\t)\n\n\t// This has to be done in the end; whether successful or not, we should mark that the recovery is done.\n\t// So that after the active period passes, we are able to run other recoveries.\n\tdefer func() {\n\t\tif err := resolveRecovery(topologyRecovery, nil); err != nil {\n\t\t\tlogger.Error(\n\t\t\t\t\"failed to resolve recovery\",\n\t\t\t\tslog.String(\"recovery\", ReconcileStaleTopoPrimaryRecoveryName),\n\t\t\t\tslog.Any(\"error\", err),\n\t\t\t)\n\t\t}\n\t}()\n\n\tanalyzedTablet, err := inst.ReadTablet(alias)\n\tif err != nil {\n\t\tlogger.Error(\"failed to read tablet, aborting recovery\", slog.String(\"tablet\", aliasString))\n\t\treturn false, topologyRecovery, fmt.Errorf(\"failed to read instance: %w\", err)\n\t}\n\n\tvar wg sync.WaitGroup\n\n\t// Make sure the best-effort steps complete or timeout before we return.\n\tdefer wg.Wait()\n\n\t// On a best-effort basis, attempt to demote the tablet and configure replication concurrently\n\t// with the topology type update below. Failures here will not fail the overall recovery.\n\twg.Go(func() {\n\t\t// Demote the tablet, forcing it to become read-only and drop pending transactions.\n\t\tif _, err := forceDemotePrimary(ctx, analyzedTablet); err != nil {\n\t\t\tlogger.Error(\"failed to demote stale primary\", slog.String(\"tablet\", aliasString), slog.Any(\"error\", err))\n\t\t\treturn\n\t\t}\n\n\t\tlogger.Info(\"successfully demoted stale primary\", slog.String(\"tablet\", aliasString))\n","sourceCodeStart":1558,"sourceCodeEnd":1594,"githubUrl":"https://github.com/vitessio/vitess/blob/01a25a7d176f94613b8d59d799f438380a8760e4/go/vt/vtorc/logic/topology_recovery.go#L1558-L1594","documentation":"At the start of executing a recovery function, VTOrc re-reads the analyzed tablet from its local DB (inst.ReadTablet) to operate on fresh data. If that read fails, the recovery is aborted and this wrapped error is returned so the failure is attributed to the recovery run instead of being silently ignored.","triggerScenarios":"Any recovery execution path reaches the analyzedTablet lookup with an alias and inst.ReadTablet returns a DB error (backend down, schema mismatch, connection limit).","commonSituations":"VTOrc backend MySQL unavailable mid-recovery; connection pool exhausted during failover storm; upgraded Vitess with old DB schema.","solutions":["Check VTOrc backend DB health and restore connectivity; the recovery loop will retry automatically","Look at the wrapped error (%w) in VTOrc logs for the concrete SQL/DB failure","Verify schema of the vtorc instance tables after upgrades (vtorc has upgrade migrations)","Re-trigger the recovery manually via `vtorc ForcedCommand` or wait for the analysis cycle"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"if err := db.PingContext(ctx); err != nil { return err }","typeGuard":null,"tryCatchPattern":"fixed, rec, err := executeCheckAndRecoverFunction(...)\nif err != nil && strings.Contains(err.Error(), \"failed to read instance\") {\n    // backend transient: let the next cycle retry\n    return nil\n}","preventionTips":["Monitor vtorc backend DB availability","Tune connection pool limits for recovery bursts","Apply schema upgrades before restarting vtorc"],"tags":["vtorc","database","topology-recovery"],"backgroundTag":"tablet-read-failed","analyzedSha":"01a25a7d176f94613b8d59d799f438380a8760e4","analyzedAt":"2026-09-01T17:28:30.605Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}