{"record":{"id":"97705c880cf5ffad","repo":"vitessio/vitess","slug":"primary-v-version-v-is-different-than-replica-v","errorCode":null,"errorMessage":"primary %v version %v is different than replica %v version %v","messagePattern":"primary (.+?) version (.+?) is different than replica (.+?) version (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"go/vt/vtctl/grpcvtctldserver/server.go","lineNumber":5843,"sourceCode":"\tversionFuncMu.Lock()\n\tdefer versionFuncMu.Unlock()\n\treturn getVersionFromTablet\n}\n\n// helper method to asynchronously get and diff a version\nfunc (s *VtctldServer) diffVersion(ctx context.Context, primaryVersion string, primaryAlias *topodatapb.TabletAlias, alias *topodatapb.TabletAlias, wg *sync.WaitGroup, er concurrency.ErrorRecorder) {\n\tdefer wg.Done()\n\tlog.Info(fmt.Sprintf(\"Gathering version for %v\", topoproto.TabletAliasString(alias)))\n\treplicaVersion, err := s.GetVersion(ctx, &vtctldatapb.GetVersionRequest{\n\t\tTabletAlias: alias,\n\t})\n\tif err != nil {\n\t\ter.RecordError(fmt.Errorf(\"unable to get version for tablet %v: %v\", alias, err))\n\t\treturn\n\t}\n\n\tif primaryVersion != replicaVersion.Version {\n\t\ter.RecordError(fmt.Errorf(\"primary %v version %v is different than replica %v version %v\", topoproto.TabletAliasString(primaryAlias), primaryVersion, topoproto.TabletAliasString(alias), replicaVersion))\n\t}\n}\n\n// ApplyKeyspaceRoutingRules is part of the vtctlservicepb.VtctldServer interface.\nfunc (s *VtctldServer) ApplyKeyspaceRoutingRules(ctx context.Context, req *vtctldatapb.ApplyKeyspaceRoutingRulesRequest) (*vtctldatapb.ApplyKeyspaceRoutingRulesResponse, error) {\n\tspan, ctx := trace.NewSpan(ctx, \"VtctldServer.ApplyKeyspaceRoutingRules\")\n\tdefer span.Finish()\n\n\tspan.Annotate(\"skip_rebuild\", req.SkipRebuild)\n\tspan.Annotate(\"rebuild_cells\", strings.Join(req.RebuildCells, \",\"))\n\n\tresp := &vtctldatapb.ApplyKeyspaceRoutingRulesResponse{}\n\n\tupdate := func() error {\n\t\treturn topotools.UpdateKeyspaceRoutingRules(ctx, s.ts, \"ApplyKeyspaceRoutingRules\",\n\t\t\tfunc(ctx context.Context, rules *map[string]string) error {\n\t\t\t\tclear(*rules)\n\t\t\t\tfor _, rule := range req.GetKeyspaceRoutingRules().Rules {","sourceCodeStart":5825,"sourceCodeEnd":5861,"githubUrl":"https://github.com/vitessio/vitess/blob/01a25a7d176f94613b8d59d799f438380a8760e4/go/vt/vtctl/grpcvtctldserver/server.go#L5825-L5861","documentation":"After fetching versions, the server compares the primary's version string with each replica's; a mismatch is recorded as \"primary %v version %v is different than replica %v version %v\". This means tablets in the shard run different vttablet builds.","triggerScenarios":"ValidateShard/ValidateKeyspace during or after a partial rolling upgrade of vttablet binaries.","commonSituations":"Rolling upgrade interrupted; one replica missed in the deploy; canary tablet running a newer build.","solutions":["Complete the rolling upgrade so all tablets in the shard run the same version.","Downgrade the outlier tablet to the shard's standard version if the upgrade must be rolled back.","Re-run validation to confirm all versions match."],"exampleFix":null,"handlingStrategy":"validation","validationCode":"versions := map[string]string{}; for _, a := range aliases { v, err := getVersion(ctx, a); if err == nil { versions[v] = topoproto.TabletAliasString(a) } }; if len(versions) > 1 { /* versions diverge — finish rollout before validation */ }","typeGuard":null,"tryCatchPattern":"if err != nil && strings.Contains(err.Error(), \"is different than replica\") { /* parse both versions; finish or roll back the upgrade, then re-validate */ }","preventionTips":["Complete rolling upgrades shard-by-shard and validate after each shard","Track deployed vttablet versions per shard in monitoring","Never leave canary tablets on a different build long-term"],"tags":["version-mismatch","upgrade","tablet","validation"],"backgroundTag":"version-mismatch","analyzedSha":"01a25a7d176f94613b8d59d799f438380a8760e4","analyzedAt":"2026-09-01T17:28:30.605Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}