{"record":{"id":"ce3bd41fc24d3448","repo":"vitessio/vitess","slug":"failed-to-update-shard-primary-record-v","errorCode":null,"errorMessage":"failed to update shard primary record: %v","messagePattern":"failed to update shard primary record: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"go/vt/vtctl/grpcvtctldserver/server.go","lineNumber":3003,"sourceCode":"\n\t// After the primary is done, we can update the shard record\n\t// (note with semi-sync, it also means at least one replica is done).\n\twgPrimary.Wait()\n\tif primaryErr != nil {\n\t\t// The primary failed, there is no way the\n\t\t// replicas will work.  So we cancel them all.\n\t\tlogger.Warningf(\"primary failed to PopulateReparentJournal, canceling replicas\")\n\t\treplCancel()\n\t\twgReplicas.Wait()\n\t\treturn fmt.Errorf(\"failed to PopulateReparentJournal on primary: %v\", primaryErr)\n\t}\n\tif !topoproto.TabletAliasEqual(shardInfo.PrimaryAlias, req.PrimaryElectTabletAlias) {\n\t\tif _, err := s.ts.UpdateShardFields(ctx, req.Keyspace, req.Shard, func(si *topo.ShardInfo) error {\n\t\t\tsi.PrimaryAlias = req.PrimaryElectTabletAlias\n\t\t\treturn nil\n\t\t}); err != nil {\n\t\t\twgReplicas.Wait()\n\t\t\treturn fmt.Errorf(\"failed to update shard primary record: %v\", err)\n\t\t}\n\t}\n\n\t// Wait for the replicas to complete. If some of them fail, we\n\t// don't want to rebuild the shard serving graph (the failure\n\t// will most likely be a timeout, and our context will be\n\t// expired, so the rebuild will fail anyway)\n\twgReplicas.Wait()\n\tif err := rec.Error(); err != nil {\n\t\treturn err\n\t}\n\n\t// Create database if necessary on the primary. replicas will get it too through\n\t// replication. Since the user called InitShardPrimary, they've told us to\n\t// assume that whatever data is on all the replicas is what they intended.\n\t// If the database doesn't exist, it means the user intends for these tablets\n\t// to begin serving with no data (i.e. first time initialization).\n\tcreateDB := \"CREATE DATABASE IF NOT EXISTS \" + sqlescape.EscapeID(topoproto.TabletDbName(primaryElectTabletInfo.Tablet))","sourceCodeStart":2985,"sourceCodeEnd":3021,"githubUrl":"https://github.com/vitessio/vitess/blob/01a25a7d176f94613b8d59d799f438380a8760e4/go/vt/vtctl/grpcvtctldserver/server.go#L2985-L3021","documentation":"After the primary succeeds and replicas are in flight, updating the shard record's PrimaryAlias in the topology failed. The code waits for replicas, then aborts with this error — the mysql-level promotion succeeded but the topo still shows the old primary.","triggerScenarios":"s.ts.UpdateShardFields fails (topo server unavailable, permission error, version conflict) during InitShardPrimary.","commonSituations":"etcd/zookeeper outage or quota exhaustion during failover; topo credential/ACL misconfiguration; concurrent shard-record update conflicting.","solutions":["Check topo server availability and retry the operation or run `vtctldclient RebuildKeyspaceGraph`","Manually correct the shard primary alias (SetShardTabletControl / topo update) to match the promoted tablet","Verify topo credentials allow shard-record writes","Run GetShard to confirm current PrimaryAlias and reconcile with the actual primary"],"exampleFix":"// before: topo points at old primary after promotion\n// after: reconcile topo with reality\nvtctldclient GetShard ks shard\nvtctldclient RebuildKeyspaceGraph ks","handlingStrategy":"try-catch","validationCode":"// pre-check topo write access\nif err := ts.UpdateShardFields(ctx, ks, shard, func(si *topo.ShardInfo) error { return nil }); err != nil {\n    return fmt.Errorf(\"no topo write access for shard: %w\", err)\n}","typeGuard":null,"tryCatchPattern":"// after update failure, reconcile topo with the actual primary\nif strings.Contains(err.Error(), \"failed to update shard primary record\") {\n    _ = reconcileShardPrimary(ctx, ks, shard, actualPrimary)\n    return rebuildKeyspaceGraph(ctx, ks)\n}","preventionTips":["Monitor topo server availability during failovers","Grant vtctld ACLs write access to shard records","Run RebuildKeyspaceGraph after any topo write failure to resync routing","Detect PrimaryAlias vs actual-primary drift with periodic checks"],"tags":["vtctld","topology","shard-record","primary"],"backgroundTag":"shard-primary-record-update-failed","analyzedSha":"01a25a7d176f94613b8d59d799f438380a8760e4","analyzedAt":"2026-09-01T17:28:30.605Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}