{"record":{"id":"0520181fd8deff36","repo":"vitessio/vitess","slug":"can-t-rebuild-serving-keyspace-while-a-migration-i","errorCode":null,"errorMessage":"can't rebuild serving keyspace while a migration is on going. TabletControls is set for partition %v","messagePattern":"can't rebuild serving keyspace while a migration is on going\\. TabletControls is set for partition (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"go/vt/topotools/rebuild_keyspace.go","lineNumber":115,"sourceCode":"\t\treturn err\n\t}\n\n\t// This is safe to rebuild as long there are not srvKeyspaces with tablet controls set.\n\t// Build the list of cells to work on: we get the union\n\t// of all the Cells of all the Shards, limited to the provided cells.\n\t//\n\t// srvKeyspaceMap is a map:\n\t//   key: cell\n\t//   value: topo.SrvKeyspace object being built\n\tsrvKeyspaceMap := make(map[string]*topodatapb.SrvKeyspace)\n\tfor _, cell := range cells {\n\t\tsrvKeyspace, err := ts.GetSrvKeyspace(ctx, cell, keyspace)\n\t\tswitch {\n\t\tcase err == nil:\n\t\t\tfor _, partition := range srvKeyspace.GetPartitions() {\n\t\t\t\tfor _, shardTabletControl := range partition.GetShardTabletControls() {\n\t\t\t\t\tif shardTabletControl.QueryServiceDisabled {\n\t\t\t\t\t\treturn fmt.Errorf(\"can't rebuild serving keyspace while a migration is on going. TabletControls is set for partition %v\", partition)\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}\n\t\tcase topo.IsErrType(err, topo.NoNode):\n\t\t\t// NOOP\n\t\tdefault:\n\t\t\treturn err\n\t\t}\n\t\tsrvKeyspaceMap[cell] = &topodatapb.SrvKeyspace{\n\t\t\tThrottlerConfig: ki.ThrottlerConfig,\n\t\t}\n\t}\n\n\tservedTypes := []topodatapb.TabletType{topodatapb.TabletType_PRIMARY, topodatapb.TabletType_REPLICA, topodatapb.TabletType_RDONLY}\n\n\t// for each entry in the srvKeyspaceMap map, we do the following:\n\t// - get the Shard structures for each shard / cell\n\t// - if not present, build an empty one from global Shard","sourceCodeStart":97,"sourceCodeEnd":133,"githubUrl":"https://github.com/vitessio/vitess/blob/01a25a7d176f94613b8d59d799f438380a8760e4/go/vt/topotools/rebuild_keyspace.go#L97-L133","documentation":"During RebuildKeyspaceLocked, before updating serving graph data, vitess checks each cell's existing SrvKeyspace for ShardTabletControls with QueryServiceDisabled set. Such controls indicate a keyspace migration (e.g. a MoveTables/Reshard switch-over or MigrateRepo-style migration) is in progress; rebuilding serving data now would clobber migration state, so the rebuild is refused.","triggerScenarios":"Running RebuildKeyspace (directly or via workflows/primitives that call it) on a keyspace whose SrvKeyspace partitions contain ShardTabletControl entries with QueryServiceDisabled=true — i.e. between the 'disable queries' and 'finalize' steps of a resharding/migration.","commonSituations":"A vtctld operator runs RebuildKeyspace while a MigrateRepo/reshard workflow is mid-flight; a previous migration crashed leaving stale QueryServiceDisabled controls; a user copied serving data from a keyspace under migration.","solutions":["Wait for the in-flight migration/reshard workflow to complete (finish its switch traffic/finalize steps) before rebuilding serving keyspace data.","Complete or clean up the interrupted migration: use vtctldclient workflow commands (or RemoveKeyspaceCopying/switch steps) to finish or abort so TabletControls are cleared.","If no migration is actually running, inspect the SrvKeyspace in each cell (vtctldclient GetSrvKeyspace) and manually clear the stale ShardTabletControls / rebuild serving data with migration state accounted for."],"exampleFix":"// before: unconditional rebuild\nts.RebuildKeyspace(ctx, cells, keyspace)\n// after: check for active tablet controls first\nsrv, _ := ts.GetSrvKeyspace(ctx, cell, keyspace)\nfor _, p := range srv.GetPartitions() {\n    for _, c := range p.GetShardTabletControls() {\n        if c.QueryServiceDisabled {\n            return fmt.Errorf(\"migration in progress for %s, aborting rebuild\", keyspace)\n        }\n    }\n}\nts.RebuildKeyspace(ctx, cells, keyspace)","handlingStrategy":"validation","validationCode":"srv, err := ts.GetSrvKeyspace(ctx, cell, keyspace)\nif err == nil {\n    for _, p := range srv.GetPartitions() {\n        for _, c := range p.GetShardTabletControls() {\n            if c.QueryServiceDisabled {\n                return fmt.Errorf(\"migration active for %v/%v; rebuild blocked\", cell, keyspace)\n            }\n        }\n    }\n}","typeGuard":"func migrationInProgress(srv *topodatapb.SrvKeyspace) bool {\n    for _, p := range srv.GetPartitions() {\n        for _, c := range p.GetShardTabletControls() {\n            if c.QueryServiceDisabled {\n                return true\n            }\n        }\n    }\n    return false\n}","tryCatchPattern":"if err := ts.RebuildKeyspace(ctx, cells, keyspace); err != nil {\n    if strings.Contains(err.Error(), \"migration is on going\") {\n        // defer rebuild until workflow finalize completes\n        return scheduleRebuildAfterWorkflow(keyspace)\n    }\n    return err\n}","preventionTips":["Never run manual RebuildKeyspace while a MoveTables/Reshard workflow is between switch-traffic and finalize","Check workflow status (vtctldclient workflow list) before touching serving graph data","Clean up aborted migrations promptly so stale TabletControls don't block later rebuilds"],"tags":["topo","rebuild-keyspace","resharding","migration"],"backgroundTag":"migration-in-progress-conflict","analyzedSha":"01a25a7d176f94613b8d59d799f438380a8760e4","analyzedAt":"2026-09-01T17:28:30.605Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}