vitessio/vitess · error

can't rebuild serving keyspace while a migration is on going

Error message

can't rebuild serving keyspace while a migration is on going. TabletControls is set for partition %v

What it means

During RebuildKeyspaceLocked, before updating serving graph data, vitess checks each cell's existing SrvKeyspace for ShardTabletControls with QueryServiceDisabled set. Such controls indicate a keyspace migration (e.g. a MoveTables/Reshard switch-over or MigrateRepo-style migration) is in progress; rebuilding serving data now would clobber migration state, so the rebuild is refused.

Source

Thrown at go/vt/topotools/rebuild_keyspace.go:115

		return err
	}

	// This is safe to rebuild as long there are not srvKeyspaces with tablet controls set.
	// Build the list of cells to work on: we get the union
	// of all the Cells of all the Shards, limited to the provided cells.
	//
	// srvKeyspaceMap is a map:
	//   key: cell
	//   value: topo.SrvKeyspace object being built
	srvKeyspaceMap := make(map[string]*topodatapb.SrvKeyspace)
	for _, cell := range cells {
		srvKeyspace, err := ts.GetSrvKeyspace(ctx, cell, keyspace)
		switch {
		case err == nil:
			for _, partition := range srvKeyspace.GetPartitions() {
				for _, shardTabletControl := range partition.GetShardTabletControls() {
					if shardTabletControl.QueryServiceDisabled {
						return fmt.Errorf("can't rebuild serving keyspace while a migration is on going. TabletControls is set for partition %v", partition)
					}
				}
			}
		case topo.IsErrType(err, topo.NoNode):
			// NOOP
		default:
			return err
		}
		srvKeyspaceMap[cell] = &topodatapb.SrvKeyspace{
			ThrottlerConfig: ki.ThrottlerConfig,
		}
	}

	servedTypes := []topodatapb.TabletType{topodatapb.TabletType_PRIMARY, topodatapb.TabletType_REPLICA, topodatapb.TabletType_RDONLY}

	// for each entry in the srvKeyspaceMap map, we do the following:
	// - get the Shard structures for each shard / cell
	// - if not present, build an empty one from global Shard

View on GitHub (pinned to 01a25a7d17)

Solutions

1. Wait for the in-flight migration/reshard workflow to complete (finish its switch traffic/finalize steps) before rebuilding serving keyspace data.
2. Complete or clean up the interrupted migration: use vtctldclient workflow commands (or RemoveKeyspaceCopying/switch steps) to finish or abort so TabletControls are cleared.
3. If no migration is actually running, inspect the SrvKeyspace in each cell (vtctldclient GetSrvKeyspace) and manually clear the stale ShardTabletControls / rebuild serving data with migration state accounted for.

Example fix

// before: unconditional rebuild
ts.RebuildKeyspace(ctx, cells, keyspace)
// after: check for active tablet controls first
srv, _ := ts.GetSrvKeyspace(ctx, cell, keyspace)
for _, p := range srv.GetPartitions() {
    for _, c := range p.GetShardTabletControls() {
        if c.QueryServiceDisabled {
            return fmt.Errorf("migration in progress for %s, aborting rebuild", keyspace)
        }
    }
}
ts.RebuildKeyspace(ctx, cells, keyspace)

srv, err := ts.GetSrvKeyspace(ctx, cell, keyspace)
if err == nil {
    for _, p := range srv.GetPartitions() {
        for _, c := range p.GetShardTabletControls() {
            if c.QueryServiceDisabled {
                return fmt.Errorf("migration active for %v/%v; rebuild blocked", cell, keyspace)
            }
        }
    }
}

func migrationInProgress(srv *topodatapb.SrvKeyspace) bool {
    for _, p := range srv.GetPartitions() {
        for _, c := range p.GetShardTabletControls() {
            if c.QueryServiceDisabled {
                return true
            }
        }
    }
    return false
}

if err := ts.RebuildKeyspace(ctx, cells, keyspace); err != nil {
    if strings.Contains(err.Error(), "migration is on going") {
        // defer rebuild until workflow finalize completes
        return scheduleRebuildAfterWorkflow(keyspace)
    }
    return err
}

When it happens

Trigger: Running RebuildKeyspace (directly or via workflows/primitives that call it) on a keyspace whose SrvKeyspace partitions contain ShardTabletControl entries with QueryServiceDisabled=true — i.e. between the 'disable queries' and 'finalize' steps of a resharding/migration.

Common situations: A vtctld operator runs RebuildKeyspace while a MigrateRepo/reshard workflow is mid-flight; a previous migration crashed leaving stale QueryServiceDisabled controls; a user copied serving data from a keyspace under migration.

Related errors

• writing serving data failed: %v
• same keyrange is present in source and target: %v
• source and target keyranges don't match: %v vs %v
• %w: uid %v is already in use
• %w: no SourceShard with uid %v

AI-assisted analysis of vitessio/vitess@01a25a7d17 (2026-09-01). Data as JSON: /api/errors/0520181fd8deff36. Report an issue: GitHub.