vitessio/vitess · critical

failed to update shard primary record: %v

Error message

failed to update shard primary record: %v

What it means

After the primary succeeds and replicas are in flight, updating the shard record's PrimaryAlias in the topology failed. The code waits for replicas, then aborts with this error — the mysql-level promotion succeeded but the topo still shows the old primary.

Source

Thrown at go/vt/vtctl/grpcvtctldserver/server.go:3003

	// After the primary is done, we can update the shard record
	// (note with semi-sync, it also means at least one replica is done).
	wgPrimary.Wait()
	if primaryErr != nil {
		// The primary failed, there is no way the
		// replicas will work.  So we cancel them all.
		logger.Warningf("primary failed to PopulateReparentJournal, canceling replicas")
		replCancel()
		wgReplicas.Wait()
		return fmt.Errorf("failed to PopulateReparentJournal on primary: %v", primaryErr)
	}
	if !topoproto.TabletAliasEqual(shardInfo.PrimaryAlias, req.PrimaryElectTabletAlias) {
		if _, err := s.ts.UpdateShardFields(ctx, req.Keyspace, req.Shard, func(si *topo.ShardInfo) error {
			si.PrimaryAlias = req.PrimaryElectTabletAlias
			return nil
		}); err != nil {
			wgReplicas.Wait()
			return fmt.Errorf("failed to update shard primary record: %v", err)
		}
	}

	// Wait for the replicas to complete. If some of them fail, we
	// don't want to rebuild the shard serving graph (the failure
	// will most likely be a timeout, and our context will be
	// expired, so the rebuild will fail anyway)
	wgReplicas.Wait()
	if err := rec.Error(); err != nil {
		return err
	}

	// Create database if necessary on the primary. replicas will get it too through
	// replication. Since the user called InitShardPrimary, they've told us to
	// assume that whatever data is on all the replicas is what they intended.
	// If the database doesn't exist, it means the user intends for these tablets
	// to begin serving with no data (i.e. first time initialization).
	createDB := "CREATE DATABASE IF NOT EXISTS " + sqlescape.EscapeID(topoproto.TabletDbName(primaryElectTabletInfo.Tablet))

View on GitHub (pinned to 01a25a7d17)

Solutions

1. Check topo server availability and retry the operation or run `vtctldclient RebuildKeyspaceGraph`
2. Manually correct the shard primary alias (SetShardTabletControl / topo update) to match the promoted tablet
3. Verify topo credentials allow shard-record writes
4. Run GetShard to confirm current PrimaryAlias and reconcile with the actual primary

Example fix

// before: topo points at old primary after promotion
// after: reconcile topo with reality
vtctldclient GetShard ks shard
vtctldclient RebuildKeyspaceGraph ks

// pre-check topo write access
if err := ts.UpdateShardFields(ctx, ks, shard, func(si *topo.ShardInfo) error { return nil }); err != nil {
    return fmt.Errorf("no topo write access for shard: %w", err)
}

// after update failure, reconcile topo with the actual primary
if strings.Contains(err.Error(), "failed to update shard primary record") {
    _ = reconcileShardPrimary(ctx, ks, shard, actualPrimary)
    return rebuildKeyspaceGraph(ctx, ks)
}

When it happens

Trigger: s.ts.UpdateShardFields fails (topo server unavailable, permission error, version conflict) during InitShardPrimary.

Common situations: etcd/zookeeper outage or quota exhaustion during failover; topo credential/ACL misconfiguration; concurrent shard-record update conflicting.

Related errors

• parse error
• no shards found in keyspace
• no local cells have been created yet
• no primary tablet found
• shard %s/%s has no primary

AI-assisted analysis of vitessio/vitess@01a25a7d17 (2026-09-01). Data as JSON: /api/errors/ce3bd41fc24d3448. Report an issue: GitHub.