primary %v version %v is different than replica %v version %

Error message

primary %v version %v is different than replica %v version %v

What it means

After fetching versions, the server compares the primary's version string with each replica's; a mismatch is recorded as "primary %v version %v is different than replica %v version %v". This means tablets in the shard run different vttablet builds.

Source

Thrown at go/vt/vtctl/grpcvtctldserver/server.go:5843

	versionFuncMu.Lock()
	defer versionFuncMu.Unlock()
	return getVersionFromTablet
}

// helper method to asynchronously get and diff a version
func (s *VtctldServer) diffVersion(ctx context.Context, primaryVersion string, primaryAlias *topodatapb.TabletAlias, alias *topodatapb.TabletAlias, wg *sync.WaitGroup, er concurrency.ErrorRecorder) {
	defer wg.Done()
	log.Info(fmt.Sprintf("Gathering version for %v", topoproto.TabletAliasString(alias)))
	replicaVersion, err := s.GetVersion(ctx, &vtctldatapb.GetVersionRequest{
		TabletAlias: alias,
	})
	if err != nil {
		er.RecordError(fmt.Errorf("unable to get version for tablet %v: %v", alias, err))
		return
	}

	if primaryVersion != replicaVersion.Version {
		er.RecordError(fmt.Errorf("primary %v version %v is different than replica %v version %v", topoproto.TabletAliasString(primaryAlias), primaryVersion, topoproto.TabletAliasString(alias), replicaVersion))
	}
}

// ApplyKeyspaceRoutingRules is part of the vtctlservicepb.VtctldServer interface.
func (s *VtctldServer) ApplyKeyspaceRoutingRules(ctx context.Context, req *vtctldatapb.ApplyKeyspaceRoutingRulesRequest) (*vtctldatapb.ApplyKeyspaceRoutingRulesResponse, error) {
	span, ctx := trace.NewSpan(ctx, "VtctldServer.ApplyKeyspaceRoutingRules")
	defer span.Finish()

	span.Annotate("skip_rebuild", req.SkipRebuild)
	span.Annotate("rebuild_cells", strings.Join(req.RebuildCells, ","))

	resp := &vtctldatapb.ApplyKeyspaceRoutingRulesResponse{}

	update := func() error {
		return topotools.UpdateKeyspaceRoutingRules(ctx, s.ts, "ApplyKeyspaceRoutingRules",
			func(ctx context.Context, rules *map[string]string) error {
				clear(*rules)
				for _, rule := range req.GetKeyspaceRoutingRules().Rules {

View on GitHub (pinned to 01a25a7d17)

Solutions

Complete the rolling upgrade so all tablets in the shard run the same version.
Downgrade the outlier tablet to the shard's standard version if the upgrade must be rolled back.
Re-run validation to confirm all versions match.

Defensive patterns

Strategy: validation

Validate before calling

versions := map[string]string{}; for _, a := range aliases { v, err := getVersion(ctx, a); if err == nil { versions[v] = topoproto.TabletAliasString(a) } }; if len(versions) > 1 { /* versions diverge — finish rollout before validation */ }

Try / catch

if err != nil && strings.Contains(err.Error(), "is different than replica") { /* parse both versions; finish or roll back the upgrade, then re-validate */ }

Prevention

Complete rolling upgrades shard-by-shard and validate after each shard
Track deployed vttablet versions per shard in monitoring
Never leave canary tablets on a different build long-term

When it happens

Trigger: ValidateShard/ValidateKeyspace during or after a partial rolling upgrade of vttablet binaries.

Common situations: Rolling upgrade interrupted; one replica missed in the deploy; canary tablet running a newer build.

Related errors

AI-assisted analysis of vitessio/vitess@01a25a7d17 (2026-09-01). Data as JSON: /api/errors/97705c880cf5ffad. Report an issue: GitHub.