What causes "failed to read primary instance %v: %w" in vitessio/vitess?

Retry on the next vtorc cycle — this is often transient right after promotion until the primary is discovered

failed to read primary instance %v: %w

Error message

failed to read primary instance %v: %w

What it means

After resolving the shard primary alias, vtorc reads that primary's full instance data with ReadInstance. If the read fails (e.g. the instance row is not yet in vtorc's backend or a DB error occurs), the error is wrapped with the primary's tablet alias. This guard prevents running errant-GTID diffing against unknown primary data.

Source

Thrown at go/vt/vtorc/inst/instance_dao.go:445

	if err != nil {
		return fmt.Errorf("failed to read shard primary for %s/%s: %w", tablet.Keyspace, tablet.Shard, err)
	}

	// Check if the current tablet is the primary. If it is, then we don't need to
	// run errant GTID detection on it.
	if topoproto.TabletAliasEqual(primaryAlias, instance.InstanceAlias) {
		// A primary cannot have errant GTIDs relative to itself; clear any
		// value left over from before this tablet was promoted.
		instance.GtidErrant = ""
		currentErrantGTIDCount.Reset(tabletAliasString)
		return nil
	}

	var primaryInstance *Instance
	if primaryAlias != nil {
		primaryInstance, _, err = ReadInstance(primaryAlias)
		if err != nil {
			return fmt.Errorf("failed to read primary instance %v: %w", topoproto.TabletAliasString(primaryAlias), err)
		}
	}

	// Only run errant GTID detection if we are sure that the data read of the current primary
	// is up-to-date enough to reflect that it has been promoted. This is needed to prevent
	// flagging incorrect errant GTIDs. If we were to use old data, we could have some GTIDs
	// accepted by the old primary (this tablet) that don't show in the new primary's set.
	if primaryInstance != nil && primaryInstance.SourceHost == "" {
		// If the instance has no replication source and no primary GTID set yet, or if the instance's replication
		// source is not the primary, use the shard primary's executed GTID set for comparison.
		if (instance.SourceHost == "" && instance.primaryExecutedGtidSet == "") || !sourceIsPrimary(instance, primaryInstance) {
			instance.primaryExecutedGtidSet = primaryInstance.ExecutedGtidSet
		}
	}

	var errantGtidCount int64
	if instance.ExecutedGtidSet != "" && instance.primaryExecutedGtidSet != "" {
		// Compare primary & replica GTID sets, but ignore the sets that present the primary's UUID.

View on GitHub (pinned to 01a25a7d17)

Solutions

Retry on the next vtorc cycle — this is often transient right after promotion until the primary is discovered
Check vtorc logs for errors reading the primary tablet; verify the primary tablet is up and registered in the topo
Inspect the wrapped cause (%w) for backend DB errors vs 'instance not found'
Force a topology refresh of the shard or restart vtorc discovery if the primary stays unread

Example fix

// diagnosis aid
// before
return fmt.Errorf("failed to read primary instance %v: %w", topoproto.TabletAliasString(primaryAlias), err)
// after (operational fix, not code): ensure new primary tablets are readable
vtctlclient GetTablet zone1-0000000100  # confirm tablet exists in topo before expecting vtorc reads to succeed

Defensive patterns

Strategy: retry

Validate before calling

// Confirm the shard primary is known to vtorc before relying on errant-GTID detection
// vtctlclient GetTablet <primary-alias>   # should succeed and show PRIMARY type

Try / catch

err := runErrantGTIDCheck(ctx, tablet)
if err != nil && strings.Contains(err.Error(), "failed to read primary instance") {
    // common right after promotion; wait for next discovery cycle and retry
    time.Sleep(discoveryInterval)
    err = runErrantGTIDCheck(ctx, tablet)
}

Prevention

Wait for post-promotion discovery to settle before trusting errant-GTID results
Keep primary tablets registered and healthy in the topo
Watch vtorc logs for repeated ReadInstance failures on the primary alias
Check vtorc's backend DB health (it stores instance data there)

When it happens

Trigger: detectErrantGTIDs calling ReadInstance(primaryAlias) when the primary has not been discovered/read yet (newly promoted primary), or the underlying backend read returns an error.

Common situations: Immediately after a failover/promotion before vtorc's discovery has read the new primary, backend MySQL (sqlite/MySQL backing vtorc's data) issues, or a primary that was decommissioned while a replica still points at it.

Related errors

AI-assisted analysis of vitessio/vitess@01a25a7d17 (2026-09-01). Data as JSON: /api/errors/e4d56b5b846337a6. Report an issue: GitHub.