{"record":{"id":"3fdc15a1071e41b8","repo":"vitessio/vitess","slug":"replicationstatus-s-failed-s","errorCode":null,"errorMessage":"ReplicationStatus(%s) failed: %s","messagePattern":"ReplicationStatus\\((.+?)\\) failed: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"go/vt/vtctl/grpcvtctldserver/server.go","lineNumber":4252,"sourceCode":"\t\t\t\tspan, ctx := trace.NewSpan(ctx, \"VtctldServer.getReplicationStatus\")\n\t\t\t\tdefer span.Finish()\n\n\t\t\t\tspan.Annotate(\"tablet_alias\", alias)\n\n\t\t\t\tctx, cancel := context.WithTimeout(ctx, topo.RemoteOperationTimeout)\n\t\t\t\tdefer cancel()\n\n\t\t\t\tstatus, err := s.tmc.ReplicationStatus(ctx, tablet)\n\t\t\t\tif err != nil {\n\t\t\t\t\tswitch ctx.Err() {\n\t\t\t\t\tcase context.Canceled:\n\t\t\t\t\t\tlog.Warn(fmt.Sprintf(\"context canceled before obtaining replication position from %s: %s\", alias, err))\n\t\t\t\t\tcase context.DeadlineExceeded:\n\t\t\t\t\t\tlog.Warn(fmt.Sprintf(\"context deadline exceeded before obtaining replication position from %s: %s\", alias, err))\n\t\t\t\t\tdefault:\n\t\t\t\t\t\t// The RPC was not timed out or canceled. We treat this\n\t\t\t\t\t\t// as a fatal error for the overall request.\n\t\t\t\t\t\trec.RecordError(fmt.Errorf(\"ReplicationStatus(%s) failed: %s\", alias, err))\n\t\t\t\t\t\treturn\n\t\t\t\t\t}\n\n\t\t\t\t\tstatus = nil // Don't record any position for this tablet.\n\t\t\t\t}\n\n\t\t\t\tm.Lock()\n\t\t\t\tdefer m.Unlock()\n\n\t\t\t\tresults[alias] = status\n\t\t\t\ttabletMap[alias] = tablet\n\t\t\t}(ctx, alias, tabletInfo.Tablet)\n\t\t}\n\t}\n\n\twg.Wait()\n\n\tif rec.HasErrors() {","sourceCodeStart":4234,"sourceCodeEnd":4270,"githubUrl":"https://github.com/vitessio/vitess/blob/01a25a7d176f94613b8d59d799f438380a8760e4/go/vt/vtctl/grpcvtctldserver/server.go#L4234-L4270","documentation":"Aggregated error recorded when fetching a replica's ReplicationStatus fails for reasons other than context cancellation or deadline exceeded during shard replication status gathering. Like the PrimaryPosition case, a non-timeout error is fatal for the whole request.","triggerScenarios":"Shard replication status request where a tablet's ReplicationStatus tabletmanager RPC fails (connection refused, tablet down, gRPC error), and the error is not Canceled or DeadlineExceeded.","commonSituations":"A replica tablet is stopped mid-request; MySQL is down on the replica so SHOW REPLICA STATUS fails; network partition between vtctld and the tablet.","solutions":["Check that all tablets in the shard are serving and MySQL is up","Verify connectivity to each tablet's tabletmanager gRPC port","Inspect the named tablet's logs for the underlying ReplicationStatus failure","Retry once the tablet recovers; replication status gathering is transient-sensitive"],"exampleFix":"// before\ndefault:\n    // The RPC was not timed out or canceled.\n    rec.RecordError(fmt.Errorf(\"ReplicationStatus(%s) failed: %s\", alias, err))\n    return\n// after\ndefault:\n    // The RPC was not timed out or canceled.\n    rec.RecordError(fmt.Errorf(\"ReplicationStatus(%s) failed: %w\", alias, err))\n    return","handlingStrategy":"retry","validationCode":"// verify replica tablets are serving before gathering statuses\nfor alias in shardTablets { _, err := tmClient.Ping(ctx, alias); if err != nil { skip(alias) } }","typeGuard":null,"tryCatchPattern":"try {\n  statuses = await client.getShardReplicationStatus(keyspace, shard)\n} catch (e) {\n  if (String(e).includes('ReplicationStatus')) {\n    // one tablet failed fatally; identify it from the error and check its health\n    checkTabletHealth(parseAlias(e)); await backoffRetry(2)\n  } else { throw e }\n}","preventionTips":["Ensure MySQL is running on every replica before status gathering","Keep all shard tablets serving; stopped tablets make status gathering fatal","Monitor replication health continuously to catch degraded replicas early","Bound requests with generous timeouts to avoid spuriously fatal non-timeout errors"],"tags":["replication","tabletmanager","grpc","replica"],"backgroundTag":"tablet-rpc-failed","analyzedSha":"01a25a7d176f94613b8d59d799f438380a8760e4","analyzedAt":"2026-09-01T17:28:30.605Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}