What causes "failed to read shard primary for %s/%s: %w" in vitessio/vitess?

Inspect the wrapped cause (%w) to see whether it is a topo connection error or 'shard not found'

failed to read shard primary for %s/%s: %w

Error message

failed to read shard primary for %s/%s: %w

What it means

During errant-GTID detection, vtorc must read the shard's current primary from the topology via ReadShardPrimaryInformation. If that topology read fails, the error is wrapped with the keyspace/shard so the operator knows which shard's primary could not be determined. Detection for that tablet is skipped.

Source

Thrown at go/vt/vtorc/inst/instance_dao.go:428

	// Something is wrong, could be network-wise. Record that we
	// tried to check the instance. last_attempted_check is also
	// updated on success by writeInstance. If the reason is a
	// stalled disk, we can record that as well.
	latency.Start("backend")
	_ = UpdateInstanceLastChecked(tabletAlias, partialSuccess, stalledDisk)
	latency.Stop("backend")
	return nil, err
}

// detectErrantGTIDs detects the errant GTIDs on an instance.
func detectErrantGTIDs(instance *Instance, tablet *topodatapb.Tablet) (err error) {
	tabletAliasString := topoproto.TabletAliasString(instance.InstanceAlias)
	// If the tablet is not replicating from anyone, then it could be the previous primary.
	// We should check for errant GTIDs by finding the difference with the shard's current primary.
	primaryAlias, _, err := ReadShardPrimaryInformation(tablet.Keyspace, tablet.Shard)
	if err != nil {
		return fmt.Errorf("failed to read shard primary for %s/%s: %w", tablet.Keyspace, tablet.Shard, err)
	}

	// Check if the current tablet is the primary. If it is, then we don't need to
	// run errant GTID detection on it.
	if topoproto.TabletAliasEqual(primaryAlias, instance.InstanceAlias) {
		// A primary cannot have errant GTIDs relative to itself; clear any
		// value left over from before this tablet was promoted.
		instance.GtidErrant = ""
		currentErrantGTIDCount.Reset(tabletAliasString)
		return nil
	}

	var primaryInstance *Instance
	if primaryAlias != nil {
		primaryInstance, _, err = ReadInstance(primaryAlias)
		if err != nil {
			return fmt.Errorf("failed to read primary instance %v: %w", topoproto.TabletAliasString(primaryAlias), err)
		}

View on GitHub (pinned to 01a25a7d17)

Solutions

Inspect the wrapped cause (%w) to see whether it is a topo connection error or 'shard not found'
Verify vtorc's topology flags (--topo_implementation, --topo_global_server_address, --topo_global_root) are correct
Check topo server health (etcd/zk) and network connectivity from the vtorc host
Confirm the keyspace/shard still exists in the topo; if resharding, wait for consistent records
The instance's health read will be retried on the next cycle once topology access recovers

Example fix

// before: opaque topo failure
// vtorc flags
--topo_implementation etcd2 --topo_global_server_address wrong-host:2379
// after
--topo_implementation etcd2 --topo_global_server_address etcd-global.example:2379 --topo_global_root /vitess/global

Defensive patterns

Strategy: retry

Validate before calling

// Pre-check topology reachability before running vtorc-dependent tooling
// vtctlclient --server vtctld:15999 GetShards <keyspace>  # must succeed from the vtorc host

Try / catch

var lastErr error
for i := 0; i < 3; i++ {
    err := runVtorcOperation(ctx)
    if err == nil {
        return nil
    }
    lastErr = err
    if !strings.Contains(err.Error(), "failed to read shard primary") {
        return err
    }
    time.Sleep(2 * time.Second)
}
return lastErr

Prevention

Monitor topo server (etcd/zk) health and alert on outages
Verify vtorc --topo_* flags against your actual topology deployment
Avoid running resharding/shard deletions while vtorc is actively scanning instances
Check the wrapped inner error (%w) to distinguish outage vs missing shard

When it happens

Trigger: detectErrantGTIDs -> ReadShardPrimaryInformation(keyscape, shard) failing due to topology (ts) unavailability, missing shard record, or stale/invalid topology data while running ReadTopologyInstanceBufferable.

Common situations: etcd/ZooKeeper outage or timeouts, shard record deleted during a reshard, vtorc pointed at the wrong topo server, or network partition between vtorc and the topology service.

Related errors

AI-assisted analysis of vitessio/vitess@01a25a7d17 (2026-09-01). Data as JSON: /api/errors/017e23d2c838e138. Report an issue: GitHub.