vitessio/vitess · error

failed to read tablet %q from vtorc db: %w

Error message

failed to read tablet %q from vtorc db: %w

What it means

During primary recovery, isPrimaryReachable checks whether the analyzed (incapacitated) primary tablet is actually reachable by first reading its Tablet record from VTOrc's local backing database (inst.ReadTablet). If that DB read fails with a non-nil error, the reachability check is aborted and this wrapped error is returned, preventing the recovery from making a decision.

Source

Thrown at go/vt/vtorc/logic/topology_recovery.go:461

	if !isERSEnabled(analysisEntry) {
		log.Warn("VTOrc not configured to run EmergencyReparentShard, skipping recovering " + recoveryName)
		return recoveryAttempted, topologyRecovery, nil
	}
	return runEmergencyReparentOp(ctx, analysisEntry, recoveryName, false, logger)
}

// isPrimaryReachable loads the analyzed primary tablet from the vtorc DB
// and checks whether tabletmanager Ping is reachable within a short timeout.
// It returns false with a nil error when the tablet is missing or incomplete so
// the caller can decide whether to skip recovery without failing outright.
func isPrimaryReachable(ctx context.Context, analysisEntry *inst.DetectionAnalysis) (bool, error) {
	if analysisEntry == nil || analysisEntry.AnalyzedInstanceAlias == nil {
		return false, nil
	}

	tablet, err := inst.ReadTablet(analysisEntry.AnalyzedInstanceAlias)
	if err != nil {
		return false, fmt.Errorf("failed to read tablet %q from vtorc db: %w", topoproto.TabletAliasString(analysisEntry.AnalyzedInstanceAlias), err)
	}

	if tablet == nil || tablet.Hostname == "" || tablet.PortMap == nil {
		return false, nil
	}

	grpcPort, ok := tablet.PortMap["grpc"]
	if !ok || grpcPort == 0 {
		return false, nil
	}

	if tmc == nil {
		return false, errors.New("tablet manager client is not initialized")
	}

	ctx, cancel := context.WithTimeout(ctx, 2*time.Second)
	defer cancel()

View on GitHub (pinned to 01a25a7d17)

Solutions

1. Check that VTOrc's backend database is up and reachable; restart VTOrc once connectivity is restored
2. Inspect VTOrc logs for the underlying wrapped error (%w) to identify the DB failure
3. Verify the inst.tablets table exists and matches the expected schema for your Vitess version (run any pending migrations)
4. Re-run recovery; the analysis loop will re-detect the problem and retry the read

// Check vtorc backend reachability first
if err := db.PingContext(ctx); err != nil { return err }

tablet, err := inst.ReadTablet(alias)
if err != nil {
    var retryable bool
    if errors.Is(err, sql.ErrConnDone) || errors.Is(err, driver.ErrBadConn) { retryable = true }
    return fmt.Errorf("failed to read tablet %q: %w (retryable=%v)", aliasStr, err, retryable)
}

When it happens

Trigger: recoverIncapacitatedPrimary calls isPrimaryReachable for an analysis entry with a non-nil AnalyzedInstanceAlias, and inst.ReadTablet(alias) returns a DB error (backend unavailable, corrupted vtorc DB, query failure).

Common situations: VTOrc's MySQL backend is down or restarting during a failover; network partition between VTOrc and its metadata store; table schema drift after a version upgrade; disk-full on the backend.

Related errors

• failed to read instance: %w
• recovery.IsRecoveryDisabled(): %v
• unmarshal primary health state: %w
• marshal primary health state: %w
• could not find primary tablet %v

AI-assisted analysis of vitessio/vitess@01a25a7d17 (2026-09-01). Data as JSON: /api/errors/3b23c8a7bd9950a0. Report an issue: GitHub.