vitessio/vitess · error

ReplicationStatus(%v) failed: %v

Error message

ReplicationStatus(%v) failed: %v

What it means

This error is recorded when an RPC to a tablet's tabletmanager (`tmc.ReplicationStatus`) fails while gathering replication status from all replicas/repl-type tablets during a reparent-related operation. It wraps the underlying gRPC error with the tablet alias so operators know which tablet could not report its status. It is collected via an error recorder, so it may appear alongside other errors rather than aborting immediately.

Source

Thrown at go/vt/vtctl/reparentutil/util.go:393

			wg.Add(1)
			go func(i int, ti *topo.TabletInfo) {
				defer wg.Done()
				pos, err := tmc.PrimaryPosition(ctx, ti.Tablet)
				if err != nil {
					rec.RecordError(fmt.Errorf("PrimaryPosition(%v) failed: %v", ti.AliasString(), err))
					return
				}
				result[i] = &replicationdatapb.Status{
					Position: pos,
				}
			}(i, ti)
		} else if ti.IsReplicaType() {
			wg.Add(1)
			go func(i int, ti *topo.TabletInfo) {
				defer wg.Done()
				status, err := tmc.ReplicationStatus(ctx, ti.Tablet)
				if err != nil {
					rec.RecordError(fmt.Errorf("ReplicationStatus(%v) failed: %v", ti.AliasString(), err))
					return
				}
				result[i] = status
			}(i, ti)
		}
	}
	wg.Wait()
	return tablets, result, rec.Error()
}

// getValidCandidatesAndPositionsAsList converts the valid candidates from a map to a list of tablets, making it easier to sort
func getValidCandidatesAndPositionsAsList(validCandidates map[string]*RelayLogPositions, tabletMap map[string]*topo.TabletInfo) ([]*topodatapb.Tablet, []*RelayLogPositions, error) {
	var validTablets []*topodatapb.Tablet
	var tabletPositions []*RelayLogPositions
	for tabletAlias, position := range validCandidates {
		tablet, isFound := tabletMap[tabletAlias]
		if !isFound {
			return nil, nil, vterrors.Errorf(vtrpc.Code_INTERNAL, "candidate %v not found in the tablet map; this an impossible situation", tabletAlias)

View on GitHub (pinned to 01a25a7d17)

Solutions

1. Check the tablet is running and healthy: `vtctldclient GetTablets` and verify the tablet service responds (`curl http://<tablet>:15000/healthz`).
2. Inspect the wrapped inner error: if connection refused, start vttablet or fix its port; if timeout, check network/firewall between vtctld and the tablet.
3. Verify mysqld is up and replication is configured on that tablet (`vtctldclient RunHealthCheck` or check /debug/vars).
4. Remove stale topo entries for dead tablets so the loop stops contacting them.

Example fix

// before: error only surfaces aggregated
rec.RecordError(fmt.Errorf("ReplicationStatus(%v) failed: %v", ti.AliasString(), err))
// after (operator action): confirm tablet reachable first
// vtctldclient GetTablets | grep <alias>; vtctldclient RunHealthCheck <alias>

// pre-check tablet reachability before reparent
for _, alias := range replicaAliases {
  ti, err := ts.GetTablet(ctx, alias)
  if err != nil || ti.Type == topodatapb.TabletType_DORMANT { log.Warnf("tablet %s unhealthy pre-check", alias) }
}

func isReplicationStatusError(err error) bool { return err != nil && strings.Contains(err.Error(), "ReplicationStatus(") }

if err := reparent(ctx, shardInfo); err != nil {
  if strings.Contains(err.Error(), "ReplicationStatus(") {
    log.Warn("a replica could not report status; check tablet health and retry")
  } else { return err }
}

When it happens

Trigger: Calling any reparent/health workflow that enumerates tablets (e.g. PlannedReparentShard, EmergencyReparentShard, or the util.GetReplicationStatuses path in reparentutil) where a tablet of type REPLICA/other IsReplicaType() tablets fails the ReplicationStatus RPC: tablet down, vttablet not listening, RPC timeout, or mysqld not running so the tablet cannot report IO/SQL thread state.

Common situations: A replica tablet is stopped or crashed mid-shard; network partition between vtctl/vtctld and the tablet; mysqld restarted so replication threads are stopped; semi-sync or lock issues making the tablet unresponsive; stale topo entries pointing at decommissioned tablets.

Related errors

• tablet %v ResetReplication failed (either fix it, or Scrap i
• can't get primary replication position: %v
• TabletExternallyReparented failed on primary %v: %v
• GetTabletMap(%v) failed: %w
• tablet %v InitReplica failed: %v

AI-assisted analysis of vitessio/vitess@01a25a7d17 (2026-09-01). Data as JSON: /api/errors/f8170739d80de12c. Report an issue: GitHub.