vitessio/vitess · error

tablet %v ResetReplication failed (either fix it, or Scrap i

Error message

tablet %v ResetReplication failed (either fix it, or Scrap it): %v

What it means

This error is recorded when the ResetReplication RPC sent to a tablet during an EmergencyReparentShard / planned reparent operation fails. It is aggregated (one entry per failing replica) and aborts the reparent, telling the operator the tablet must be fixed or Scrap-ed. The error is a wrap of the underlying tabletmanager RPC failure.

Source

Thrown at go/vt/vtctl/grpcvtctldserver/server.go:2919

	// First phase: reset replication on all tablets. If anyone fails,
	// we stop. It is probably because it is unreachable, and may leave
	// an unstable database process in the mix, with a database daemon
	// at a wrong replication spot.

	// Create a context for the following RPCs that respects waitReplicasTimeout
	resetCtx, resetCancel := context.WithTimeout(ctx, waitReplicasTimeout)
	defer resetCancel()

	event.DispatchUpdate(ev, "resetting replication on all tablets")
	wg := sync.WaitGroup{}
	rec := concurrency.AllErrorRecorder{}
	for alias, tabletInfo := range tabletMap {
		wg.Add(1)
		go func(alias string, tabletInfo *topo.TabletInfo) {
			defer wg.Done()
			logger.Infof("resetting replication on tablet %v", alias)
			if err := tmc.ResetReplication(resetCtx, tabletInfo.Tablet); err != nil {
				rec.RecordError(fmt.Errorf("tablet %v ResetReplication failed (either fix it, or Scrap it): %v", alias, err))
			}
		}(alias, tabletInfo)
	}
	wg.Wait()
	if err := rec.Error(); err != nil {
		// if any of the replicas failed
		return err
	}

	// Check we still have the topology lock.
	if err := topo.CheckShardLocked(ctx, req.Keyspace, req.Shard); err != nil {
		return fmt.Errorf("lost topology lock, aborting: %v", err)
	}

	// Tell the new primary to break its replicas, return its replication
	// position
	logger.Infof("initializing primary on %v", topoproto.TabletAliasString(req.PrimaryElectTabletAlias))
	event.DispatchUpdate(ev, "initializing primary")

View on GitHub (pinned to 01a25a7d17)

Solutions

1. Log into the failing tablet and check vttablet/mysqld health, then retry the reparent
2. Run RESET SLAVE ALL / RESET REPLICA manually on the affected tablet and retry
3. If the tablet is permanently broken, run `vtctldclient ScrapTablet` to remove it and reparent again
4. Use -force / EmergencyReparentShard if the old primary is unreachable and blocking

Example fix

// before: reparent fails on dead replica
tem.EmergencyReparentShard(ctx, keyspace, shard)
// after: scrap the broken tablet first
vtctldclient ScrapTablet <alias>
vtctldclient EmergencyReparentShard <keyspace/shard>

// check tablet reachability before reparent
for _, a := range shardTablets {
    if err := vtctld.GetTablet(ctx, a); err != nil {
        return fmt.Errorf("tablet %s unreachable before reparent: %w", a, err)
    }
}

// aggregate reparent errors and retry after fixing tablets
if err := tem.EmergencyReparentShard(ctx, ks, shard); err != nil {
    if strings.Contains(err.Error(), "ResetReplication failed") {
        // fix or scrap the named tablet, then retry once
    }
}

When it happens

Trigger: InitShardPrimary/EmergencyReparentShard via vtctld while one or more replicas fail tmc.ResetReplication — e.g. tablet unreachable, mysql down, or semi-sync settings blocked replication reset.

Common situations: A replica is stopped/decommissioned but still in the shard record; mysqld is down on a lagging replica; network partition between vtctld and the tablet during a failover; stuck replication that cannot be stopped/reset.

Related errors

• tablet %v InitReplica failed: %v
• ReplicationStatus(%v) failed: %v
• invalid key:value pair
• can't get primary replication position: %v
• failed to delete tablet: %w

AI-assisted analysis of vitessio/vitess@01a25a7d17 (2026-09-01). Data as JSON: /api/errors/a0be1ea03634507e. Report an issue: GitHub.