vitessio/vitess · error

shard is still serving

Error message

shard %s is still serving

What it means

During workflow validation (check for whether a migration has fully completed, e.g. before finalizing a shard migration), this check walks each migration source shard and records an error if the shard's primary is still marked IsPrimaryServing. A shard that still serves traffic is not safe to consider migrated/decommissioned. The error is recorded per shard via an AllErrorRecorder inside ForAllSources.

Solutions

1. Complete the traffic switch (SwitchTraffic) for the source shards so serving is disabled before finalizing.
2. Check the shard record (`vtctldclient GetTablets` / topo) and confirm IsPrimaryServing=false on migrated source primaries.
3. If serving was already drained but the flag is stale, refresh/repair the topo record via the proper vtctldclient commands rather than finalizing the workflow.

// verify source primaries are no longer serving before completing the workflow
for _, s := range sourceShards {
    if shardPrimaryInfo(ks, s).IsPrimaryServing {
        return fmt.Errorf("shard %s still serving; run SwitchTraffic first", s)
    }
}

When it happens

Trigger: Running validateWorkflowHasCompleted (traffic validation for MigrateTables/Reshard of type SHARDS) while the source shards' primaries still have IsPrimaryServing=true in the topo — i.e. PRS/traffic switch away from those shards has not happened.

Common situations: Attempting to complete a ReshardWorkflow before SwitchTraffic drained reads/writes; a failed or skipped traffic switch step; manual topo edits leaving the serving flag set.

Related errors

• mismatched shards for keyspace
• switch command shards not in topo
• topo shards not in switch command
• Create ReshardWorkflow failed
• failed setting value for

AI-assisted analysis of vitessio/vitess@01a25a7d17 (2026-09-01). Data as JSON: /api/errors/bb7fa39685c765bc. Report an issue: GitHub.

Appendix: source

Thrown at go/vt/vtctl/workflow/utils.go:558

	hasher := fnv.New64()
	hasher.Write([]byte(targetKeyspace))

	for _, s := range expanded {
		hasher.Write([]byte(s))
	}

	// Convert to int64 after dropping the highest bit.
	return int64(hasher.Sum64() & math.MaxInt64)
}

func doValidateWorkflowHasCompleted(ctx context.Context, ts *trafficSwitcher) error {
	wg := sync.WaitGroup{}
	rec := concurrency.AllErrorRecorder{}
	if ts.MigrationType() == binlogdatapb.MigrationType_SHARDS {
		_ = ts.ForAllSources(func(source *MigrationSource) error {
			wg.Add(1)
			if source.GetShard().IsPrimaryServing {
				rec.RecordError(fmt.Errorf("shard %s is still serving", source.GetShard().ShardName()))
			}
			wg.Done()
			return nil
		})
	} else {
		_ = ts.ForAllTargets(func(target *MigrationTarget) error {
			wg.Add(1)
			defer wg.Done()
			res, err := ts.ws.tmc.ReadVReplicationWorkflow(ctx, target.GetPrimary().Tablet, &tabletmanagerdatapb.ReadVReplicationWorkflowRequest{
				Workflow: ts.WorkflowName(),
			})
			if err != nil {
				rec.RecordError(err)
				return nil
			}
			for _, stream := range res.Streams {
				if stream.Message != Frozen {
					rec.RecordError(fmt.Errorf("vreplication streams are not frozen on tablet %d", target.GetPrimary().Alias.Uid))

View on GitHub (pinned to 01a25a7d17)