vitessio/vitess · error

PrimaryPosition(%v) failed: %v

Error message

PrimaryPosition(%v) failed: %v

What it means

During reparent operations, PrimaryPosition collects each current PRIMARY tablet's replication position in parallel via tmc.PrimaryPosition; any tablet RPC error is recorded as a multierr so the caller sees which primary failed and why. It indicates the tablet manager RPC to a shard's primary failed, so positions cannot be compared.

Source

Thrown at go/vt/vtctl/reparentutil/util.go:380

		return nil, nil, err
	}
	tablets := maps.Values(tabletMap)

	log.Info(fmt.Sprintf("Gathering tablet replication status for: %v", tablets))
	wg := sync.WaitGroup{}
	rec := concurrency.AllErrorRecorder{}
	result := make([]*replicationdatapb.Status, len(tablets))

	for i, ti := range tablets {
		// Don't scan tablets that won't return something
		// useful. Otherwise, you'll end up waiting for a timeout.
		if ti.Type == topodatapb.TabletType_PRIMARY {
			wg.Add(1)
			go func(i int, ti *topo.TabletInfo) {
				defer wg.Done()
				pos, err := tmc.PrimaryPosition(ctx, ti.Tablet)
				if err != nil {
					rec.RecordError(fmt.Errorf("PrimaryPosition(%v) failed: %v", ti.AliasString(), err))
					return
				}
				result[i] = &replicationdatapb.Status{
					Position: pos,
				}
			}(i, ti)
		} else if ti.IsReplicaType() {
			wg.Add(1)
			go func(i int, ti *topo.TabletInfo) {
				defer wg.Done()
				status, err := tmc.ReplicationStatus(ctx, ti.Tablet)
				if err != nil {
					rec.RecordError(fmt.Errorf("ReplicationStatus(%v) failed: %v", ti.AliasString(), err))
					return
				}
				result[i] = status
			}(i, ti)
		}

View on GitHub (pinned to 01a25a7d17)

Solutions

1. Check the named tablet is healthy: vtctldclient GetTablet <alias> and verify vttablet process/grpcs port
2. Confirm the primary's gRPC port is reachable from vtctld (telnet/nc to tablet port)
3. Re-run the reparent once the tablet is reachable; ERS tolerates and reports per-tablet errors in aggregate
4. If the tablet is stale/dead, remove its topo record or wait for ERS to proceed with reachable primaries per policy

Example fix

// before
# ERS fails: PrimaryPosition(zone1-0000000100) failed: rpc timeout
// after
systemctl status vttablet  # restart dead tablet
vtctldclient EmergencyReparentShard ks/shard

// Verify each primary tablet is reachable before gathering positions
for _, t := range primaryTablets {
	if err := tmc.Ping(ctx, t.Tablet); err != nil {
		log.Warning("primary unreachable before reparent", slog.String("tablet", t.AliasString()), slog.Any("error", err))
	}
}

err := reparentUtil(...)
if err != nil && strings.Contains(err.Error(), "PrimaryPosition") {
	// extract tablet alias, check vttablet health/gRPC port, then retry the reparent
}

When it happens

Trigger: Reparent flows (e.g., ERS position gathering) where tm.PrimaryPosition RPC to a PRIMARY tablet fails: vttablet down, grpc timeout, tablet in wrong state, or network partition between vtctld and the tablet.

Common situations: Primary vttablet crashed or is restarting mid-reparent; firewall/network drop to one cell; vttablet gRPC not enabled (vtctld cannot reach tablet manager); stale topo entry for a decommissioned primary.

Related errors

• invalid key:value pair
• not implemented in vtcombo
• send reply error: %v
• can't get primary replication position: %v
• can't dial mysqlctld: %v

AI-assisted analysis of vitessio/vitess@01a25a7d17 (2026-09-01). Data as JSON: /api/errors/d17901e84e592c88. Report an issue: GitHub.