vitessio/vitess · error

unmarshal primary health state: %w

Error message

unmarshal primary health state: %w

What it means

vtorc persists the primary health window as prototext in the primary_health table. When reading it back, prototext.Unmarshal failed on the stored string, so the stored state is corrupt or written by an incompatible schema/version. The scan error is wrapped with context.

Source

Thrown at go/vt/vtorc/inst/primary_health.go:253

// without immediately triggering recovery.
func primaryHealthWindow() time.Duration {
	return topo.RemoteOperationTimeout * 5
}

// readPrimaryHealthState loads the persisted health window for a tablet alias.
// It returns nil when no row exists, and wraps any unmarshaling errors with
// context to help identify corrupt or incompatible state.
func readPrimaryHealthState(tabletAlias string) (*primaryHealthState, error) {
	query := "select health_state from primary_health where alias = ?"
	var state *primaryHealthState
	err := db.QueryVTOrc(query, sqlutils.Args(tabletAlias), func(row sqlutils.RowMap) error {
		value := row.GetString("health_state")
		if value == "" {
			return nil
		}
		pb := &vtorcdata.PrimaryHealthState{}
		if err := prototext.Unmarshal([]byte(value), pb); err != nil {
			return fmt.Errorf("unmarshal primary health state: %w", err)
		}
		state = fromProtoPrimaryHealthState(pb)
		return nil
	})
	if err != nil {
		return nil, err
	}
	return state, nil
}

// writePrimaryHealthState persists the current health window for a tablet alias.
// It is a no-op for empty aliases or nil state, and it deletes the row if the
// state is already evictable.
func writePrimaryHealthState(tabletAlias string, state *primaryHealthState) error {
	if tabletAlias == "" || state == nil {
		return nil
	}
	if shouldEvictPrimaryHealthWindow(state) {

View on GitHub (pinned to 01a25a7d17)

Solutions

1. Inspect the offending row: SELECT alias, health_state FROM primary_health WHERE alias = '<tablet>'
2. Delete the corrupt row (DELETE FROM primary_health WHERE alias=...) — vtorc will rebuild the window from subsequent checks
3. Check for a version mismatch: if you recently upgraded vtorc, ensure the table contents were written by a compatible version
4. Restart vtorc after cleanup; the state repopulates automatically as health checks are recorded

Example fix

-- before: corrupt stored state causes unmarshal failure
SELECT health_state FROM primary_health WHERE alias='zone1-0000000100';  -- garbage
-- after: reset it
DELETE FROM primary_health WHERE alias='zone1-0000000100';

-- Detect corrupt rows before they break vtorc
SELECT alias, health_state FROM primary_health
WHERE health_state IS NOT NULL AND health_state != ''
  AND health_state NOT LIKE 'alias:%';  -- sanity heuristic: prototext of PrimaryHealthState starts with field names

state, err := readPrimaryHealthState(ctx, alias)
if err != nil {
    // corrupt/unreadable stored state: fall back to a fresh window
    log.Warn("primary health state unreadable, resetting", slog.Any("error", err))
    _ = deletePrimaryHealthState(alias)
    state = newPrimaryHealthState()
}

When it happens

Trigger: Reading the health_state column (GetString("health_state")) containing a non-empty value that is not valid prototext for vtorcdata.PrimaryHealthState — e.g. truncated row, manual edits, or format changed between Vitess versions.

Common situations: Upgrading/downgrading Vitess where the PrimaryHealthState proto changed, DBA hand-editing the primary_health table, disk/storage corruption, or an aborted write leaving partial data.

Related errors

• marshal primary health state: %w
• recovery.IsRecoveryDisabled(): %v
• failed to read tablet %q from vtorc db: %w
• failed to read instance: %w
• InsertRecoveryDetection: no detection_id returned for alias=

AI-assisted analysis of vitessio/vitess@01a25a7d17 (2026-09-01). Data as JSON: /api/errors/e64e3bd4a6266e83. Report an issue: GitHub.