{"record":{"id":"017e23d2c838e138","repo":"vitessio/vitess","slug":"failed-to-read-shard-primary-for-s-s-w","errorCode":null,"errorMessage":"failed to read shard primary for %s/%s: %w","messagePattern":"failed to read shard primary for (.+?)/(.+?): %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"go/vt/vtorc/inst/instance_dao.go","lineNumber":428,"sourceCode":"\n\t// Something is wrong, could be network-wise. Record that we\n\t// tried to check the instance. last_attempted_check is also\n\t// updated on success by writeInstance. If the reason is a\n\t// stalled disk, we can record that as well.\n\tlatency.Start(\"backend\")\n\t_ = UpdateInstanceLastChecked(tabletAlias, partialSuccess, stalledDisk)\n\tlatency.Stop(\"backend\")\n\treturn nil, err\n}\n\n// detectErrantGTIDs detects the errant GTIDs on an instance.\nfunc detectErrantGTIDs(instance *Instance, tablet *topodatapb.Tablet) (err error) {\n\ttabletAliasString := topoproto.TabletAliasString(instance.InstanceAlias)\n\t// If the tablet is not replicating from anyone, then it could be the previous primary.\n\t// We should check for errant GTIDs by finding the difference with the shard's current primary.\n\tprimaryAlias, _, err := ReadShardPrimaryInformation(tablet.Keyspace, tablet.Shard)\n\tif err != nil {\n\t\treturn fmt.Errorf(\"failed to read shard primary for %s/%s: %w\", tablet.Keyspace, tablet.Shard, err)\n\t}\n\n\t// Check if the current tablet is the primary. If it is, then we don't need to\n\t// run errant GTID detection on it.\n\tif topoproto.TabletAliasEqual(primaryAlias, instance.InstanceAlias) {\n\t\t// A primary cannot have errant GTIDs relative to itself; clear any\n\t\t// value left over from before this tablet was promoted.\n\t\tinstance.GtidErrant = \"\"\n\t\tcurrentErrantGTIDCount.Reset(tabletAliasString)\n\t\treturn nil\n\t}\n\n\tvar primaryInstance *Instance\n\tif primaryAlias != nil {\n\t\tprimaryInstance, _, err = ReadInstance(primaryAlias)\n\t\tif err != nil {\n\t\t\treturn fmt.Errorf(\"failed to read primary instance %v: %w\", topoproto.TabletAliasString(primaryAlias), err)\n\t\t}","sourceCodeStart":410,"sourceCodeEnd":446,"githubUrl":"https://github.com/vitessio/vitess/blob/01a25a7d176f94613b8d59d799f438380a8760e4/go/vt/vtorc/inst/instance_dao.go#L410-L446","documentation":"During errant-GTID detection, vtorc must read the shard's current primary from the topology via ReadShardPrimaryInformation. If that topology read fails, the error is wrapped with the keyspace/shard so the operator knows which shard's primary could not be determined. Detection for that tablet is skipped.","triggerScenarios":"detectErrantGTIDs -> ReadShardPrimaryInformation(keyscape, shard) failing due to topology (ts) unavailability, missing shard record, or stale/invalid topology data while running ReadTopologyInstanceBufferable.","commonSituations":"etcd/ZooKeeper outage or timeouts, shard record deleted during a reshard, vtorc pointed at the wrong topo server, or network partition between vtorc and the topology service.","solutions":["Inspect the wrapped cause (%w) to see whether it is a topo connection error or 'shard not found'","Verify vtorc's topology flags (--topo_implementation, --topo_global_server_address, --topo_global_root) are correct","Check topo server health (etcd/zk) and network connectivity from the vtorc host","Confirm the keyspace/shard still exists in the topo; if resharding, wait for consistent records","The instance's health read will be retried on the next cycle once topology access recovers"],"exampleFix":"// before: opaque topo failure\n// vtorc flags\n--topo_implementation etcd2 --topo_global_server_address wrong-host:2379\n// after\n--topo_implementation etcd2 --topo_global_server_address etcd-global.example:2379 --topo_global_root /vitess/global","handlingStrategy":"retry","validationCode":"// Pre-check topology reachability before running vtorc-dependent tooling\n// vtctlclient --server vtctld:15999 GetShards <keyspace>  # must succeed from the vtorc host","typeGuard":null,"tryCatchPattern":"var lastErr error\nfor i := 0; i < 3; i++ {\n    err := runVtorcOperation(ctx)\n    if err == nil {\n        return nil\n    }\n    lastErr = err\n    if !strings.Contains(err.Error(), \"failed to read shard primary\") {\n        return err\n    }\n    time.Sleep(2 * time.Second)\n}\nreturn lastErr","preventionTips":["Monitor topo server (etcd/zk) health and alert on outages","Verify vtorc --topo_* flags against your actual topology deployment","Avoid running resharding/shard deletions while vtorc is actively scanning instances","Check the wrapped inner error (%w) to distinguish outage vs missing shard"],"tags":["vtorc","topology","errant-gtid","shard"],"backgroundTag":"topology-read-failed","analyzedSha":"01a25a7d176f94613b8d59d799f438380a8760e4","analyzedAt":"2026-09-01T17:28:30.605Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}