{"record":{"id":"008e2f425ea4165b","repo":"vitessio/vitess","slug":"lost-topology-lock-aborting-v","errorCode":null,"errorMessage":"lost topology lock, aborting: %v","messagePattern":"lost topology lock, aborting: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"go/vt/vtctl/grpcvtctldserver/server.go","lineNumber":2931,"sourceCode":"\tfor alias, tabletInfo := range tabletMap {\n\t\twg.Add(1)\n\t\tgo func(alias string, tabletInfo *topo.TabletInfo) {\n\t\t\tdefer wg.Done()\n\t\t\tlogger.Infof(\"resetting replication on tablet %v\", alias)\n\t\t\tif err := tmc.ResetReplication(resetCtx, tabletInfo.Tablet); err != nil {\n\t\t\t\trec.RecordError(fmt.Errorf(\"tablet %v ResetReplication failed (either fix it, or Scrap it): %v\", alias, err))\n\t\t\t}\n\t\t}(alias, tabletInfo)\n\t}\n\twg.Wait()\n\tif err := rec.Error(); err != nil {\n\t\t// if any of the replicas failed\n\t\treturn err\n\t}\n\n\t// Check we still have the topology lock.\n\tif err := topo.CheckShardLocked(ctx, req.Keyspace, req.Shard); err != nil {\n\t\treturn fmt.Errorf(\"lost topology lock, aborting: %v\", err)\n\t}\n\n\t// Tell the new primary to break its replicas, return its replication\n\t// position\n\tlogger.Infof(\"initializing primary on %v\", topoproto.TabletAliasString(req.PrimaryElectTabletAlias))\n\tevent.DispatchUpdate(ev, \"initializing primary\")\n\trp, err := tmc.InitPrimary(ctx, primaryElectTabletInfo.Tablet, policy.SemiSyncAckers(durability, primaryElectTabletInfo.Tablet) > 0)\n\tif err != nil {\n\t\treturn err\n\t}\n\n\t// Check we stil have the topology lock.\n\tif err := topo.CheckShardLocked(ctx, req.Keyspace, req.Shard); err != nil {\n\t\treturn fmt.Errorf(\"lost topology lock, aborting: %v\", err)\n\t}\n\n\t// Create a cancelable context for the following RPCs.\n\t// If error conditions happen, we can cancel all outgoing RPCs.","sourceCodeStart":2913,"sourceCodeEnd":2949,"githubUrl":"https://github.com/vitessio/vitess/blob/01a25a7d176f94613b8d59d799f438380a8760e4/go/vt/vtctl/grpcvtctldserver/server.go#L2913-L2949","documentation":"After ResetReplication of replicas succeeds, PlannedReparentShard-style initialization re-checks that the caller still holds the shard lock in the topology server. If the lock was lost (session expired, topology flapped, or lock stolen), the operation aborts to prevent two concurrent topology-changing operations from racing.","triggerScenarios":"topo.CheckShardLocked fails between phases of InitShardPrimary — lock lease expired, topo server (etcd/zk) connection issues, or another process took the shard lock.","commonSituations":"Long-running reparent exceeding the lock timeout; etcd/zookeeper leader election or network blip during the operation; two operators running conflicting vtctld commands concurrently.","solutions":["Re-run the command once the shard is idle and lockable; it is safe to retry","Check topo server health and clocks (etcd/zk) for session drops","Increase the topo lock timeout if operations routinely exceed it","Ensure only one reparent operation runs per shard (script serialization)"],"exampleFix":"// before\nvtctldclient PlannedReparentShard ks/shard  // lock lost mid-way\n// after: retry after verifying no other reparent is running\nvtctldclient GetShard ks shard  # confirm state\nvtctldclient PlannedReparentShard ks/shard","handlingStrategy":"retry","validationCode":"// verify no concurrent shard operations before starting\nif locked, _ := topo.CheckShardLocked(ctx, ks, shard); locked {\n    return errors.New(\"shard already locked by another operation\")\n}","typeGuard":null,"tryCatchPattern":"// retry shard operations that lost their lock\nif strings.Contains(err.Error(), \"lost topology lock\") {\n    time.Sleep(lockRetryDelay)\n    return retryInitShardPrimary(ctx, req)\n}","preventionTips":["Serialize reparent operations per shard in automation (mutex/queue)","Keep shard operations short to stay within topo lock timeouts","Monitor topo server (etcd/zk) session stability and clock skew","Use one control plane (single vtctld) for topology mutations"],"tags":["vtctld","topology","shard-lock","concurrency"],"backgroundTag":"lost-topology-lock","analyzedSha":"01a25a7d176f94613b8d59d799f438380a8760e4","analyzedAt":"2026-09-01T17:28:30.605Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}