{"record":{"id":"f8170739d80de12c","repo":"vitessio/vitess","slug":"replicationstatus-v-failed-v","errorCode":null,"errorMessage":"ReplicationStatus(%v) failed: %v","messagePattern":"ReplicationStatus\\((.+?)\\) failed: (.+?)","errorType":"console","errorClass":null,"httpStatus":null,"severity":"error","filePath":"go/vt/vtctl/reparentutil/util.go","lineNumber":393,"sourceCode":"\t\t\twg.Add(1)\n\t\t\tgo func(i int, ti *topo.TabletInfo) {\n\t\t\t\tdefer wg.Done()\n\t\t\t\tpos, err := tmc.PrimaryPosition(ctx, ti.Tablet)\n\t\t\t\tif err != nil {\n\t\t\t\t\trec.RecordError(fmt.Errorf(\"PrimaryPosition(%v) failed: %v\", ti.AliasString(), err))\n\t\t\t\t\treturn\n\t\t\t\t}\n\t\t\t\tresult[i] = &replicationdatapb.Status{\n\t\t\t\t\tPosition: pos,\n\t\t\t\t}\n\t\t\t}(i, ti)\n\t\t} else if ti.IsReplicaType() {\n\t\t\twg.Add(1)\n\t\t\tgo func(i int, ti *topo.TabletInfo) {\n\t\t\t\tdefer wg.Done()\n\t\t\t\tstatus, err := tmc.ReplicationStatus(ctx, ti.Tablet)\n\t\t\t\tif err != nil {\n\t\t\t\t\trec.RecordError(fmt.Errorf(\"ReplicationStatus(%v) failed: %v\", ti.AliasString(), err))\n\t\t\t\t\treturn\n\t\t\t\t}\n\t\t\t\tresult[i] = status\n\t\t\t}(i, ti)\n\t\t}\n\t}\n\twg.Wait()\n\treturn tablets, result, rec.Error()\n}\n\n// getValidCandidatesAndPositionsAsList converts the valid candidates from a map to a list of tablets, making it easier to sort\nfunc getValidCandidatesAndPositionsAsList(validCandidates map[string]*RelayLogPositions, tabletMap map[string]*topo.TabletInfo) ([]*topodatapb.Tablet, []*RelayLogPositions, error) {\n\tvar validTablets []*topodatapb.Tablet\n\tvar tabletPositions []*RelayLogPositions\n\tfor tabletAlias, position := range validCandidates {\n\t\ttablet, isFound := tabletMap[tabletAlias]\n\t\tif !isFound {\n\t\t\treturn nil, nil, vterrors.Errorf(vtrpc.Code_INTERNAL, \"candidate %v not found in the tablet map; this an impossible situation\", tabletAlias)","sourceCodeStart":375,"sourceCodeEnd":411,"githubUrl":"https://github.com/vitessio/vitess/blob/01a25a7d176f94613b8d59d799f438380a8760e4/go/vt/vtctl/reparentutil/util.go#L375-L411","documentation":"This error is recorded when an RPC to a tablet's tabletmanager (`tmc.ReplicationStatus`) fails while gathering replication status from all replicas/repl-type tablets during a reparent-related operation. It wraps the underlying gRPC error with the tablet alias so operators know which tablet could not report its status. It is collected via an error recorder, so it may appear alongside other errors rather than aborting immediately.","triggerScenarios":"Calling any reparent/health workflow that enumerates tablets (e.g. PlannedReparentShard, EmergencyReparentShard, or the util.GetReplicationStatuses path in reparentutil) where a tablet of type REPLICA/other IsReplicaType() tablets fails the ReplicationStatus RPC: tablet down, vttablet not listening, RPC timeout, or mysqld not running so the tablet cannot report IO/SQL thread state.","commonSituations":"A replica tablet is stopped or crashed mid-shard; network partition between vtctl/vtctld and the tablet; mysqld restarted so replication threads are stopped; semi-sync or lock issues making the tablet unresponsive; stale topo entries pointing at decommissioned tablets.","solutions":["Check the tablet is running and healthy: `vtctldclient GetTablets` and verify the tablet service responds (`curl http://<tablet>:15000/healthz`).","Inspect the wrapped inner error: if connection refused, start vttablet or fix its port; if timeout, check network/firewall between vtctld and the tablet.","Verify mysqld is up and replication is configured on that tablet (`vtctldclient RunHealthCheck` or check /debug/vars).","Remove stale topo entries for dead tablets so the loop stops contacting them."],"exampleFix":"// before: error only surfaces aggregated\nrec.RecordError(fmt.Errorf(\"ReplicationStatus(%v) failed: %v\", ti.AliasString(), err))\n// after (operator action): confirm tablet reachable first\n// vtctldclient GetTablets | grep <alias>; vtctldclient RunHealthCheck <alias>","handlingStrategy":"try-catch","validationCode":"// pre-check tablet reachability before reparent\nfor _, alias := range replicaAliases {\n  ti, err := ts.GetTablet(ctx, alias)\n  if err != nil || ti.Type == topodatapb.TabletType_DORMANT { log.Warnf(\"tablet %s unhealthy pre-check\", alias) }\n}","typeGuard":"func isReplicationStatusError(err error) bool { return err != nil && strings.Contains(err.Error(), \"ReplicationStatus(\") }","tryCatchPattern":"if err := reparent(ctx, shardInfo); err != nil {\n  if strings.Contains(err.Error(), \"ReplicationStatus(\") {\n    log.Warn(\"a replica could not report status; check tablet health and retry\")\n  } else { return err }\n}","preventionTips":["Monitor tablet health endpoints so dead tablets are caught before reparent operations.","Prune stale tablets from the topo regularly.","Run reparents with generous but bounded RPC timeouts.","Alert on mysqld restarts that leave replication threads stopped."],"tags":["replication","grpc","tablet","reparent"],"backgroundTag":"tablet-rpc-failed","analyzedSha":"01a25a7d176f94613b8d59d799f438380a8760e4","analyzedAt":"2026-09-01T17:28:30.605Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}