vitessio/vitess · error

lost topology lock, aborting: %v

Error message

lost topology lock, aborting: %v

What it means

After ResetReplication of replicas succeeds, PlannedReparentShard-style initialization re-checks that the caller still holds the shard lock in the topology server. If the lock was lost (session expired, topology flapped, or lock stolen), the operation aborts to prevent two concurrent topology-changing operations from racing.

Source

Thrown at go/vt/vtctl/grpcvtctldserver/server.go:2931

	for alias, tabletInfo := range tabletMap {
		wg.Add(1)
		go func(alias string, tabletInfo *topo.TabletInfo) {
			defer wg.Done()
			logger.Infof("resetting replication on tablet %v", alias)
			if err := tmc.ResetReplication(resetCtx, tabletInfo.Tablet); err != nil {
				rec.RecordError(fmt.Errorf("tablet %v ResetReplication failed (either fix it, or Scrap it): %v", alias, err))
			}
		}(alias, tabletInfo)
	}
	wg.Wait()
	if err := rec.Error(); err != nil {
		// if any of the replicas failed
		return err
	}

	// Check we still have the topology lock.
	if err := topo.CheckShardLocked(ctx, req.Keyspace, req.Shard); err != nil {
		return fmt.Errorf("lost topology lock, aborting: %v", err)
	}

	// Tell the new primary to break its replicas, return its replication
	// position
	logger.Infof("initializing primary on %v", topoproto.TabletAliasString(req.PrimaryElectTabletAlias))
	event.DispatchUpdate(ev, "initializing primary")
	rp, err := tmc.InitPrimary(ctx, primaryElectTabletInfo.Tablet, policy.SemiSyncAckers(durability, primaryElectTabletInfo.Tablet) > 0)
	if err != nil {
		return err
	}

	// Check we stil have the topology lock.
	if err := topo.CheckShardLocked(ctx, req.Keyspace, req.Shard); err != nil {
		return fmt.Errorf("lost topology lock, aborting: %v", err)
	}

	// Create a cancelable context for the following RPCs.
	// If error conditions happen, we can cancel all outgoing RPCs.

View on GitHub (pinned to 01a25a7d17)

Solutions

1. Re-run the command once the shard is idle and lockable; it is safe to retry
2. Check topo server health and clocks (etcd/zk) for session drops
3. Increase the topo lock timeout if operations routinely exceed it
4. Ensure only one reparent operation runs per shard (script serialization)

Example fix

// before
vtctldclient PlannedReparentShard ks/shard  // lock lost mid-way
// after: retry after verifying no other reparent is running
vtctldclient GetShard ks shard  # confirm state
vtctldclient PlannedReparentShard ks/shard

// verify no concurrent shard operations before starting
if locked, _ := topo.CheckShardLocked(ctx, ks, shard); locked {
    return errors.New("shard already locked by another operation")
}

// retry shard operations that lost their lock
if strings.Contains(err.Error(), "lost topology lock") {
    time.Sleep(lockRetryDelay)
    return retryInitShardPrimary(ctx, req)
}

When it happens

Trigger: topo.CheckShardLocked fails between phases of InitShardPrimary — lock lease expired, topo server (etcd/zk) connection issues, or another process took the shard lock.

Common situations: Long-running reparent exceeding the lock timeout; etcd/zookeeper leader election or network blip during the operation; two operators running conflicting vtctld commands concurrently.

Related errors

• invalid choice for enum
• value must be either a float64 (interpreted as seconds) or a
• flagutil: NewOptionalFlag requires a non-nil parse function
• flagutil: OptionalFlagValue has no parse function; use a con
• parse error

AI-assisted analysis of vitessio/vitess@01a25a7d17 (2026-09-01). Data as JSON: /api/errors/008e2f425ea4165b. Report an issue: GitHub.