vitessio/vitess · error

shard %s is still serving

Error message

shard %s is still serving

What it means

During workflow validation (check for whether a migration has fully completed, e.g. before finalizing a shard migration), this check walks each migration source shard and records an error if the shard's primary is still marked IsPrimaryServing. A shard that still serves traffic is not safe to consider migrated/decommissioned. The error is recorded per shard via an AllErrorRecorder inside ForAllSources.

Source

Thrown at go/vt/vtctl/workflow/utils.go:558

	hasher := fnv.New64()
	hasher.Write([]byte(targetKeyspace))

	for _, s := range expanded {
		hasher.Write([]byte(s))
	}

	// Convert to int64 after dropping the highest bit.
	return int64(hasher.Sum64() & math.MaxInt64)
}

func doValidateWorkflowHasCompleted(ctx context.Context, ts *trafficSwitcher) error {
	wg := sync.WaitGroup{}
	rec := concurrency.AllErrorRecorder{}
	if ts.MigrationType() == binlogdatapb.MigrationType_SHARDS {
		_ = ts.ForAllSources(func(source *MigrationSource) error {
			wg.Add(1)
			if source.GetShard().IsPrimaryServing {
				rec.RecordError(fmt.Errorf("shard %s is still serving", source.GetShard().ShardName()))
			}
			wg.Done()
			return nil
		})
	} else {
		_ = ts.ForAllTargets(func(target *MigrationTarget) error {
			wg.Add(1)
			defer wg.Done()
			res, err := ts.ws.tmc.ReadVReplicationWorkflow(ctx, target.GetPrimary().Tablet, &tabletmanagerdatapb.ReadVReplicationWorkflowRequest{
				Workflow: ts.WorkflowName(),
			})
			if err != nil {
				rec.RecordError(err)
				return nil
			}
			for _, stream := range res.Streams {
				if stream.Message != Frozen {
					rec.RecordError(fmt.Errorf("vreplication streams are not frozen on tablet %d", target.GetPrimary().Alias.Uid))

View on GitHub (pinned to 01a25a7d17)

Solutions

1. Complete the traffic switch (SwitchTraffic) for the source shards so serving is disabled before finalizing.
2. Check the shard record (`vtctldclient GetTablets` / topo) and confirm IsPrimaryServing=false on migrated source primaries.
3. If serving was already drained but the flag is stale, refresh/repair the topo record via the proper vtctldclient commands rather than finalizing the workflow.

// verify source primaries are no longer serving before completing the workflow
for _, s := range sourceShards {
    if shardPrimaryInfo(ks, s).IsPrimaryServing {
        return fmt.Errorf("shard %s still serving; run SwitchTraffic first", s)
    }
}

When it happens

Trigger: Running validateWorkflowHasCompleted (traffic validation for MigrateTables/Reshard of type SHARDS) while the source shards' primaries still have IsPrimaryServing=true in the topo — i.e. PRS/traffic switch away from those shards has not happened.

Common situations: Attempting to complete a ReshardWorkflow before SwitchTraffic drained reads/writes; a failed or skipped traffic switch step; manual topo edits leaving the serving flag set.

Related errors

• switch command shards not in topo: %v
• topo shards not in switch command: %v
• mismatched shards for keyspace %s: %s
• range %d should be >= %d
• UnescapeID err: invalid input identifier '%s'

AI-assisted analysis of vitessio/vitess@01a25a7d17 (2026-09-01). Data as JSON: /api/errors/bb7fa39685c765bc. Report an issue: GitHub.