{"record":{"id":"111a7c87039ec08f","repo":"vitessio/vitess","slug":"failed-to-populatereparentjournal-on-primary-v","errorCode":null,"errorMessage":"failed to PopulateReparentJournal on primary: %v","messagePattern":"failed to PopulateReparentJournal on primary: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"go/vt/vtctl/grpcvtctldserver/server.go","lineNumber":2995,"sourceCode":"\t\t\t\tdefer wgReplicas.Done()\n\t\t\t\tlogger.Infof(\"initializing replica %v\", alias)\n\t\t\t\tif err := tmc.InitReplica(replCtx, tabletInfo.Tablet, req.PrimaryElectTabletAlias, rp, now, policy.IsReplicaSemiSync(durability, primaryElectTabletInfo.Tablet, tabletInfo.Tablet)); err != nil {\n\t\t\t\t\trec.RecordError(fmt.Errorf(\"tablet %v InitReplica failed: %v\", alias, err))\n\t\t\t\t}\n\t\t\t}(alias, tabletInfo)\n\t\t}\n\t}\n\n\t// After the primary is done, we can update the shard record\n\t// (note with semi-sync, it also means at least one replica is done).\n\twgPrimary.Wait()\n\tif primaryErr != nil {\n\t\t// The primary failed, there is no way the\n\t\t// replicas will work.  So we cancel them all.\n\t\tlogger.Warningf(\"primary failed to PopulateReparentJournal, canceling replicas\")\n\t\treplCancel()\n\t\twgReplicas.Wait()\n\t\treturn fmt.Errorf(\"failed to PopulateReparentJournal on primary: %v\", primaryErr)\n\t}\n\tif !topoproto.TabletAliasEqual(shardInfo.PrimaryAlias, req.PrimaryElectTabletAlias) {\n\t\tif _, err := s.ts.UpdateShardFields(ctx, req.Keyspace, req.Shard, func(si *topo.ShardInfo) error {\n\t\t\tsi.PrimaryAlias = req.PrimaryElectTabletAlias\n\t\t\treturn nil\n\t\t}); err != nil {\n\t\t\twgReplicas.Wait()\n\t\t\treturn fmt.Errorf(\"failed to update shard primary record: %v\", err)\n\t\t}\n\t}\n\n\t// Wait for the replicas to complete. If some of them fail, we\n\t// don't want to rebuild the shard serving graph (the failure\n\t// will most likely be a timeout, and our context will be\n\t// expired, so the rebuild will fail anyway)\n\twgReplicas.Wait()\n\tif err := rec.Error(); err != nil {\n\t\treturn err","sourceCodeStart":2977,"sourceCodeEnd":3013,"githubUrl":"https://github.com/vitessio/vitess/blob/01a25a7d176f94613b8d59d799f438380a8760e4/go/vt/vtctl/grpcvtctldserver/server.go#L2977-L3013","documentation":"The newly promoted primary failed to write its reparent journal (a marker record proving it is the authoritative primary). Since replicas synchronize on this journal, the operation cancels all replica RPCs and aborts with the primary's underlying error.","triggerScenarios":"InitShardPrimary: tmc.PopulateReparentJournal on the promoted primary returns an error — primary tablet RPC failed, mysql write failed, or the tablet was reset mid-operation.","commonSituations":"Primary mysqld read-only or crashed right after promotion; tablet agent restarted during reparent; disk/full or replication log errors on the new primary.","solutions":["Inspect the new primary tablet logs for why the journal write failed and fix mysql health","Retry InitShardPrimary once the primary is reachable and writable","Verify the primary is not read-only (super_read_only / read_only flags)","Fall back to EmergencyReparentShard or manual promotion if primary state is inconsistent"],"exampleFix":"// before\nprimary> SHOW VARIABLES LIKE 'read_only';  -- ON\n// after\nprimary> SET GLOBAL read_only=0; SET GLOBAL super_read_only=0;\nvtctldclient PlannedReparentShard ks/shard","handlingStrategy":"try-catch","validationCode":"// ensure the candidate primary is writable before promotion\nif ro := mysqlShowVariable(primary, \"read_only\"); ro == \"ON\" {\n    return errors.New(\"candidate primary is read_only=ON\")\n}","typeGuard":null,"tryCatchPattern":"// abort cleanly and inspect primary on journal failure\nif strings.Contains(err.Error(), \"failed to PopulateReparentJournal\") {\n    replCtxCancel()\n    return inspectPrimaryAndRetry(ctx, newPrimary)\n}","preventionTips":["Verify new primary mysqld is writable (read_only off) before failover","Alert on primary tablet agent restarts during maintenance windows","Ensure adequate disk space and no replication-log errors on primaries","Test reparent-journal behavior in drills"],"tags":["vtctld","reparent","primary","reparent-journal"],"backgroundTag":"populate-reparent-journal-failed","analyzedSha":"01a25a7d176f94613b8d59d799f438380a8760e4","analyzedAt":"2026-09-01T17:28:30.605Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}