vitessio/vitess · critical

failed to PopulateReparentJournal on primary: %v

Error message

failed to PopulateReparentJournal on primary: %v

What it means

The newly promoted primary failed to write its reparent journal (a marker record proving it is the authoritative primary). Since replicas synchronize on this journal, the operation cancels all replica RPCs and aborts with the primary's underlying error.

Source

Thrown at go/vt/vtctl/grpcvtctldserver/server.go:2995

				defer wgReplicas.Done()
				logger.Infof("initializing replica %v", alias)
				if err := tmc.InitReplica(replCtx, tabletInfo.Tablet, req.PrimaryElectTabletAlias, rp, now, policy.IsReplicaSemiSync(durability, primaryElectTabletInfo.Tablet, tabletInfo.Tablet)); err != nil {
					rec.RecordError(fmt.Errorf("tablet %v InitReplica failed: %v", alias, err))
				}
			}(alias, tabletInfo)
		}
	}

	// After the primary is done, we can update the shard record
	// (note with semi-sync, it also means at least one replica is done).
	wgPrimary.Wait()
	if primaryErr != nil {
		// The primary failed, there is no way the
		// replicas will work.  So we cancel them all.
		logger.Warningf("primary failed to PopulateReparentJournal, canceling replicas")
		replCancel()
		wgReplicas.Wait()
		return fmt.Errorf("failed to PopulateReparentJournal on primary: %v", primaryErr)
	}
	if !topoproto.TabletAliasEqual(shardInfo.PrimaryAlias, req.PrimaryElectTabletAlias) {
		if _, err := s.ts.UpdateShardFields(ctx, req.Keyspace, req.Shard, func(si *topo.ShardInfo) error {
			si.PrimaryAlias = req.PrimaryElectTabletAlias
			return nil
		}); err != nil {
			wgReplicas.Wait()
			return fmt.Errorf("failed to update shard primary record: %v", err)
		}
	}

	// Wait for the replicas to complete. If some of them fail, we
	// don't want to rebuild the shard serving graph (the failure
	// will most likely be a timeout, and our context will be
	// expired, so the rebuild will fail anyway)
	wgReplicas.Wait()
	if err := rec.Error(); err != nil {
		return err

View on GitHub (pinned to 01a25a7d17)

Solutions

1. Inspect the new primary tablet logs for why the journal write failed and fix mysql health
2. Retry InitShardPrimary once the primary is reachable and writable
3. Verify the primary is not read-only (super_read_only / read_only flags)
4. Fall back to EmergencyReparentShard or manual promotion if primary state is inconsistent

Example fix

// before
primary> SHOW VARIABLES LIKE 'read_only';  -- ON
// after
primary> SET GLOBAL read_only=0; SET GLOBAL super_read_only=0;
vtctldclient PlannedReparentShard ks/shard

// ensure the candidate primary is writable before promotion
if ro := mysqlShowVariable(primary, "read_only"); ro == "ON" {
    return errors.New("candidate primary is read_only=ON")
}

// abort cleanly and inspect primary on journal failure
if strings.Contains(err.Error(), "failed to PopulateReparentJournal") {
    replCtxCancel()
    return inspectPrimaryAndRetry(ctx, newPrimary)
}

When it happens

Trigger: InitShardPrimary: tmc.PopulateReparentJournal on the promoted primary returns an error — primary tablet RPC failed, mysql write failed, or the tablet was reset mid-operation.

Common situations: Primary mysqld read-only or crashed right after promotion; tablet agent restarted during reparent; disk/full or replication log errors on the new primary.

Related errors

• tablet %v ResetReplication failed (either fix it, or Scrap i
• tablet %v InitReplica failed: %v
• failed to update shard primary record: %v
• failed to create database: %v
• no primary in shard %v/%v

AI-assisted analysis of vitessio/vitess@01a25a7d17 (2026-09-01). Data as JSON: /api/errors/111a7c87039ec08f. Report an issue: GitHub.