{"record":{"id":"d17901e84e592c88","repo":"vitessio/vitess","slug":"primaryposition-v-failed-v","errorCode":null,"errorMessage":"PrimaryPosition(%v) failed: %v","messagePattern":"PrimaryPosition\\((.+?)\\) failed: (.+?)","errorType":"console","errorClass":null,"httpStatus":null,"severity":"error","filePath":"go/vt/vtctl/reparentutil/util.go","lineNumber":380,"sourceCode":"\t\treturn nil, nil, err\n\t}\n\ttablets := maps.Values(tabletMap)\n\n\tlog.Info(fmt.Sprintf(\"Gathering tablet replication status for: %v\", tablets))\n\twg := sync.WaitGroup{}\n\trec := concurrency.AllErrorRecorder{}\n\tresult := make([]*replicationdatapb.Status, len(tablets))\n\n\tfor i, ti := range tablets {\n\t\t// Don't scan tablets that won't return something\n\t\t// useful. Otherwise, you'll end up waiting for a timeout.\n\t\tif ti.Type == topodatapb.TabletType_PRIMARY {\n\t\t\twg.Add(1)\n\t\t\tgo func(i int, ti *topo.TabletInfo) {\n\t\t\t\tdefer wg.Done()\n\t\t\t\tpos, err := tmc.PrimaryPosition(ctx, ti.Tablet)\n\t\t\t\tif err != nil {\n\t\t\t\t\trec.RecordError(fmt.Errorf(\"PrimaryPosition(%v) failed: %v\", ti.AliasString(), err))\n\t\t\t\t\treturn\n\t\t\t\t}\n\t\t\t\tresult[i] = &replicationdatapb.Status{\n\t\t\t\t\tPosition: pos,\n\t\t\t\t}\n\t\t\t}(i, ti)\n\t\t} else if ti.IsReplicaType() {\n\t\t\twg.Add(1)\n\t\t\tgo func(i int, ti *topo.TabletInfo) {\n\t\t\t\tdefer wg.Done()\n\t\t\t\tstatus, err := tmc.ReplicationStatus(ctx, ti.Tablet)\n\t\t\t\tif err != nil {\n\t\t\t\t\trec.RecordError(fmt.Errorf(\"ReplicationStatus(%v) failed: %v\", ti.AliasString(), err))\n\t\t\t\t\treturn\n\t\t\t\t}\n\t\t\t\tresult[i] = status\n\t\t\t}(i, ti)\n\t\t}","sourceCodeStart":362,"sourceCodeEnd":398,"githubUrl":"https://github.com/vitessio/vitess/blob/01a25a7d176f94613b8d59d799f438380a8760e4/go/vt/vtctl/reparentutil/util.go#L362-L398","documentation":"During reparent operations, PrimaryPosition collects each current PRIMARY tablet's replication position in parallel via tmc.PrimaryPosition; any tablet RPC error is recorded as a multierr so the caller sees which primary failed and why. It indicates the tablet manager RPC to a shard's primary failed, so positions cannot be compared.","triggerScenarios":"Reparent flows (e.g., ERS position gathering) where tm.PrimaryPosition RPC to a PRIMARY tablet fails: vttablet down, grpc timeout, tablet in wrong state, or network partition between vtctld and the tablet.","commonSituations":"Primary vttablet crashed or is restarting mid-reparent; firewall/network drop to one cell; vttablet gRPC not enabled (vtctld cannot reach tablet manager); stale topo entry for a decommissioned primary.","solutions":["Check the named tablet is healthy: vtctldclient GetTablet <alias> and verify vttablet process/grpcs port","Confirm the primary's gRPC port is reachable from vtctld (telnet/nc to tablet port)","Re-run the reparent once the tablet is reachable; ERS tolerates and reports per-tablet errors in aggregate","If the tablet is stale/dead, remove its topo record or wait for ERS to proceed with reachable primaries per policy"],"exampleFix":"// before\n# ERS fails: PrimaryPosition(zone1-0000000100) failed: rpc timeout\n// after\nsystemctl status vttablet  # restart dead tablet\nvtctldclient EmergencyReparentShard ks/shard","handlingStrategy":"retry","validationCode":"// Verify each primary tablet is reachable before gathering positions\nfor _, t := range primaryTablets {\n\tif err := tmc.Ping(ctx, t.Tablet); err != nil {\n\t\tlog.Warning(\"primary unreachable before reparent\", slog.String(\"tablet\", t.AliasString()), slog.Any(\"error\", err))\n\t}\n}","typeGuard":null,"tryCatchPattern":"err := reparentUtil(...)\nif err != nil && strings.Contains(err.Error(), \"PrimaryPosition\") {\n\t// extract tablet alias, check vttablet health/gRPC port, then retry the reparent\n}","preventionTips":["Monitor vttablet gRPC reachability from vtctld per cell","Ensure tablet gRPC is enabled in all cells","Verify primary topo records are not stale before reparents","Use graceful reparent windows avoiding tablet restarts"],"tags":["reparent","tablet-manager","rpc"],"backgroundTag":"primary-position-rpc-failed","analyzedSha":"01a25a7d176f94613b8d59d799f438380a8760e4","analyzedAt":"2026-09-01T17:28:30.605Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}