{"record":{"id":"4580ba9ebe59408f","repo":"vitessio/vitess","slug":"can-t-get-primary-replication-position-v","errorCode":null,"errorMessage":"can't get primary replication position: %v","messagePattern":"can't get primary replication position: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"go/vt/mysqlctl/builtinbackupengine.go","lineNumber":1841,"sourceCode":"}\n\nfunc (be *BuiltinBackupEngine) Name() string { return builtinBackupEngineName }\n\nfunc getPrimaryPosition(ctx context.Context, tmc tmclient.TabletManagerClient, ts *topo.Server, keyspace, shard string) (replication.Position, error) {\n\tsi, err := ts.GetShard(ctx, keyspace, shard)\n\tif err != nil {\n\t\treturn replication.Position{}, vterrors.Wrap(err, \"can't read shard\")\n\t}\n\tif topoproto.TabletAliasIsZero(si.PrimaryAlias) {\n\t\treturn replication.Position{}, fmt.Errorf(\"shard %v/%v has no primary\", keyspace, shard)\n\t}\n\tti, err := ts.GetTablet(ctx, si.PrimaryAlias)\n\tif err != nil {\n\t\treturn replication.Position{}, fmt.Errorf(\"can't get primary tablet record %v: %v\", topoproto.TabletAliasString(si.PrimaryAlias), err)\n\t}\n\tposStr, err := tmc.PrimaryPosition(ctx, ti.Tablet)\n\tif err != nil {\n\t\treturn replication.Position{}, fmt.Errorf(\"can't get primary replication position: %v\", err)\n\t}\n\tpos, err := replication.DecodePosition(posStr)\n\tif err != nil {\n\t\treturn replication.Position{}, fmt.Errorf(\"can't decode primary replication position %q: %v\", posStr, err)\n\t}\n\treturn pos, nil\n}\n\nfunc init() {\n\tBackupRestoreEngineMap[builtinBackupEngineName] = &BuiltinBackupEngine{}\n}\n\n// closeWithRetry does just what it says. Retrying a close operation is important as\n// an error is most likely transient/ephemeral and leaving around open file descriptors\n// can lead to later problems as the file may be in a sort-of uploaded state where it\n// exists but has not yet been finalized (this is true for GCS). This can cause\n// unexpected behavior if you retry the file while the original request is still in this\n// state. Most implementations such as GCS will automatically retry operations, but close","sourceCodeStart":1823,"sourceCodeEnd":1859,"githubUrl":"https://github.com/vitessio/vitess/blob/01a25a7d176f94613b8d59d799f438380a8760e4/go/vt/mysqlctl/builtinbackupengine.go#L1823-L1859","documentation":"getPrimaryPosition calls tmc.PrimaryPosition on the primary tablet to get its current replication position. If the TabletManager RPC fails, this error wraps the cause. It means Vitess reached (or tried to reach) the primary tablet but could not obtain its position.","triggerScenarios":"tmc.PrimaryPosition(ctx, ti.Tablet) returns an error — tablet agent down, RPC timeout, TLS/auth mismatch between vtctld/vtworker and the tablet, or mysqld unreachable from the tablet agent.","commonSituations":"Primary tablet process restarting during backup; network partition or firewall blocking the tablet's gRPC port; tablet agent cannot query mysqld for the position.","solutions":["Check the primary tablet's health and gRPC port (vtctldclient GetTablet, then hit /debug vars).","Verify connectivity/auth (certs, grpc port) between the caller and the tablet.","Confirm mysqld is running on the primary so the agent can read SHOW REPLICA STATUS / binlog position.","Retry once the tablet is healthy — the failure is often transient."],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// check tablet reachability before the RPC\nif err := checkTabletHealth(ctx, ti.Tablet); err != nil {\n\treturn fmt.Errorf(\"primary tablet unhealthy, skipping position fetch: %w\", err)\n}","typeGuard":null,"tryCatchPattern":"pos, err := getPrimaryPosition(ctx, tmc, ts, ks, shard)\nif err != nil && strings.Contains(err.Error(), \"replication position\") {\n\t// transient tablet RPC failure: retry with backoff\n\tpos, err = retryWithBackoff(ctx, func() (replication.Position, error) {\n\t\treturn getPrimaryPosition(ctx, tmc, ts, ks, shard)\n\t})\n}","preventionTips":["Monitor tablet gRPC port reachability and health checks.","Keep TLS/auth configs consistent between clients and tablets.","Avoid backups while the primary is restarting or under heavy load."],"tags":["rpc","tablet","replication"],"backgroundTag":"tablet-rpc-failed","analyzedSha":"01a25a7d176f94613b8d59d799f438380a8760e4","analyzedAt":"2026-09-01T17:28:30.605Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}