{"record":{"id":"db322ba0cbb356d8","repo":"vitessio/vitess","slug":"failed-to-ensure-replication-was-started-on-tablet","errorCode":null,"errorMessage":"failed to ensure replication was started on tablet %s after RestartReplication error (%v): %w","messagePattern":"failed to ensure replication was started on tablet (.+?) after RestartReplication error \\((.+?)\\): %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"go/vt/vtorc/logic/topology_recovery.go","lineNumber":666,"sourceCode":"\t// when the RPC never reached the tablet, in which case STOP cannot have run.\n\tif ctx.Err() != nil || status.Code(err) == codes.Unavailable {\n\t\treturn err\n\t}\n\n\t// TODO: Remove this StartReplication fallback in v26, when all supported\n\t// vttablets include detached post-STOP cleanup in RestartReplication.\n\ttabletAlias := topoproto.TabletAliasString(tablet.Alias)\n\tlogger.Warn(\"RestartReplication failed; attempting to ensure replication is started\",\n\t\tslog.String(\"tablet\", tabletAlias),\n\t\tslog.Any(\"error\", err),\n\t)\n\n\tstartCtx, startCancel := context.WithTimeout(context.WithoutCancel(ctx), topo.RemoteOperationTimeout)\n\tdefer startCancel()\n\tif startErr := tmc.StartReplication(startCtx, tablet, semiSync); startErr != nil {\n\t\t// Wrap with %w (not vterrors.Wrapf, which has no Unwrap) so callers can\n\t\t// still match on the original RestartReplication error.\n\t\treturn fmt.Errorf(\"failed to ensure replication was started on tablet %s after RestartReplication error (%v): %w\", tabletAlias, startErr, err)\n\t}\n\n\treturn err\n}\n\n// isERSEnabled returns true if ERS can be used globally or for the given keyspace.\nfunc isERSEnabled(analysisEntry *inst.DetectionAnalysis) bool {\n\t// If ERS is disabled globally we have no way of repairing the cluster.\n\tif !config.ERSEnabled() {\n\t\tlog.Info(fmt.Sprintf(\"VTOrc not configured to run ERS, skipping recovering %v\", analysisEntry.Analysis))\n\t\treturn false\n\t}\n\n\t// Return false if ERS is disabled on the keyspace.\n\tif analysisEntry.AnalyzedKeyspaceEmergencyReparentDisabled {\n\t\tlog.Info(fmt.Sprintf(\"ERS is disabled on keyspace %s, skipping recovering %v\", analysisEntry.AnalyzedKeyspace, analysisEntry.Analysis))\n\t\treturn false\n\t}","sourceCodeStart":648,"sourceCodeEnd":684,"githubUrl":"https://github.com/vitessio/vitess/blob/01a25a7d176f94613b8d59d799f438380a8760e4/go/vt/vtorc/logic/topology_recovery.go#L648-L684","documentation":"restartReplication issues RestartReplication on a tablet and, if that call fails, attempts a compensating StartReplication to ensure replication is at least running. If StartReplication also fails, the two errors are combined into this wrapped error: the outer message names the tablet and the StartReplication failure, and %w preserves the original RestartReplication error so callers can still match on it.","triggerScenarios":"A replication restart during recovery fails (err != nil), the fallback tmc.StartReplication startErr is also non-nil — typically both fail because the tablet is unreachable, the tablet's MySQL is down, or semi-sync settings conflict.","commonSituations":"Recovering after a primary failure where replicas are also down or partitioned; mysqld not running on the replica; tabletmanager RPC auth/TLS misconfiguration; semi-sync replica count impossible after primary loss.","solutions":["Read the wrapped (%w) original RestartReplication error plus the startErr to see the root cause","Confirm the replica tablet and its mysqld are actually up (vtctldclient GetTablet / check the tablet's logs)","Manually run START REPLICA on the replica's MySQL to restore replication, then re-run recovery","Check semi-sync configuration — after primary loss, replicas may block; adjust rpl_semi_sync settings or disable semi-sync temporarily"],"exampleFix":"// root cause visible in the wrapped error\nif err != nil {\n    var orig *mysql.SQLError\n    if errors.As(err, &orig) { /* handle original RestartReplication error */ }\n}","handlingStrategy":"try-catch","validationCode":"// Probe tablet reachability before restart\nerr := tmc.Ping(ctx, tablet)","typeGuard":null,"tryCatchPattern":"err := restartReplication(ctx, tablet, semiSync)\nif err != nil {\n    orig := err\n    if unwrapped := errors.Unwrap(err); unwrapped != nil { orig = unwrapped }\n    log.Error(\"replication restart failed\", slog.Any(\"error\", orig))\n}","preventionTips":["Ensure replicas' mysqld is up before initiating ERS/replication restarts","Review semi-sync settings that can block START REPLICA after primary loss","Verify tabletmanager RPC TLS/credentials"],"tags":["vtorc","replication","tabletmanager"],"backgroundTag":"replication-restart-failed","analyzedSha":"01a25a7d176f94613b8d59d799f438380a8760e4","analyzedAt":"2026-09-01T17:28:30.605Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}