{"record":{"id":"a0be1ea03634507e","repo":"vitessio/vitess","slug":"tablet-v-resetreplication-failed-either-fix-it","errorCode":null,"errorMessage":"tablet %v ResetReplication failed (either fix it, or Scrap it): %v","messagePattern":"tablet (.+?) ResetReplication failed \\(either fix it, or Scrap it\\): (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"go/vt/vtctl/grpcvtctldserver/server.go","lineNumber":2919,"sourceCode":"\t// First phase: reset replication on all tablets. If anyone fails,\n\t// we stop. It is probably because it is unreachable, and may leave\n\t// an unstable database process in the mix, with a database daemon\n\t// at a wrong replication spot.\n\n\t// Create a context for the following RPCs that respects waitReplicasTimeout\n\tresetCtx, resetCancel := context.WithTimeout(ctx, waitReplicasTimeout)\n\tdefer resetCancel()\n\n\tevent.DispatchUpdate(ev, \"resetting replication on all tablets\")\n\twg := sync.WaitGroup{}\n\trec := concurrency.AllErrorRecorder{}\n\tfor alias, tabletInfo := range tabletMap {\n\t\twg.Add(1)\n\t\tgo func(alias string, tabletInfo *topo.TabletInfo) {\n\t\t\tdefer wg.Done()\n\t\t\tlogger.Infof(\"resetting replication on tablet %v\", alias)\n\t\t\tif err := tmc.ResetReplication(resetCtx, tabletInfo.Tablet); err != nil {\n\t\t\t\trec.RecordError(fmt.Errorf(\"tablet %v ResetReplication failed (either fix it, or Scrap it): %v\", alias, err))\n\t\t\t}\n\t\t}(alias, tabletInfo)\n\t}\n\twg.Wait()\n\tif err := rec.Error(); err != nil {\n\t\t// if any of the replicas failed\n\t\treturn err\n\t}\n\n\t// Check we still have the topology lock.\n\tif err := topo.CheckShardLocked(ctx, req.Keyspace, req.Shard); err != nil {\n\t\treturn fmt.Errorf(\"lost topology lock, aborting: %v\", err)\n\t}\n\n\t// Tell the new primary to break its replicas, return its replication\n\t// position\n\tlogger.Infof(\"initializing primary on %v\", topoproto.TabletAliasString(req.PrimaryElectTabletAlias))\n\tevent.DispatchUpdate(ev, \"initializing primary\")","sourceCodeStart":2901,"sourceCodeEnd":2937,"githubUrl":"https://github.com/vitessio/vitess/blob/01a25a7d176f94613b8d59d799f438380a8760e4/go/vt/vtctl/grpcvtctldserver/server.go#L2901-L2937","documentation":"This error is recorded when the ResetReplication RPC sent to a tablet during an EmergencyReparentShard / planned reparent operation fails. It is aggregated (one entry per failing replica) and aborts the reparent, telling the operator the tablet must be fixed or Scrap-ed. The error is a wrap of the underlying tabletmanager RPC failure.","triggerScenarios":"InitShardPrimary/EmergencyReparentShard via vtctld while one or more replicas fail tmc.ResetReplication — e.g. tablet unreachable, mysql down, or semi-sync settings blocked replication reset.","commonSituations":"A replica is stopped/decommissioned but still in the shard record; mysqld is down on a lagging replica; network partition between vtctld and the tablet during a failover; stuck replication that cannot be stopped/reset.","solutions":["Log into the failing tablet and check vttablet/mysqld health, then retry the reparent","Run RESET SLAVE ALL / RESET REPLICA manually on the affected tablet and retry","If the tablet is permanently broken, run `vtctldclient ScrapTablet` to remove it and reparent again","Use -force / EmergencyReparentShard if the old primary is unreachable and blocking"],"exampleFix":"// before: reparent fails on dead replica\ntem.EmergencyReparentShard(ctx, keyspace, shard)\n// after: scrap the broken tablet first\nvtctldclient ScrapTablet <alias>\nvtctldclient EmergencyReparentShard <keyspace/shard>","handlingStrategy":"retry","validationCode":"// check tablet reachability before reparent\nfor _, a := range shardTablets {\n    if err := vtctld.GetTablet(ctx, a); err != nil {\n        return fmt.Errorf(\"tablet %s unreachable before reparent: %w\", a, err)\n    }\n}","typeGuard":null,"tryCatchPattern":"// aggregate reparent errors and retry after fixing tablets\nif err := tem.EmergencyReparentShard(ctx, ks, shard); err != nil {\n    if strings.Contains(err.Error(), \"ResetReplication failed\") {\n        // fix or scrap the named tablet, then retry once\n    }\n}","preventionTips":["Monitor vttablet/mysqld health on all replicas before planned failovers","Scrap decommissioned tablets promptly so they don't block reparents","Use EmergencyReparentShard -force only after inspecting tablet states","Alert on replication stop/reset failures from vttablet logs"],"tags":["vtctld","reparent","replication","tablet"],"backgroundTag":"reset-replication-failed","analyzedSha":"01a25a7d176f94613b8d59d799f438380a8760e4","analyzedAt":"2026-09-01T17:28:30.605Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}