What causes "vreplication streams are not frozen on tablet %d" in vitessio/vitess?

Wait until the streams are caught up and the freeze step marks them Frozen (check `_vt.vreplication` message on the target primary), then retry validation.

vitessio/vitess · error

vreplication streams are not frozen on tablet %d

Error message

vreplication streams are not frozen on tablet %d

What it means

When validating that a workflow has completed for targets of a SHARDS-type migration, this check queries vreplication streams on each target's primary tablet and expects every stream's message to equal the sentinel Frozen. Any stream whose message is not Frozen means the target has not been frozen as part of migration completion, so this error is recorded with the tablet's alias UID.

Source

Thrown at go/vt/vtctl/workflow/utils.go:576

				rec.RecordError(fmt.Errorf("shard %s is still serving", source.GetShard().ShardName()))
			}
			wg.Done()
			return nil
		})
	} else {
		_ = ts.ForAllTargets(func(target *MigrationTarget) error {
			wg.Add(1)
			defer wg.Done()
			res, err := ts.ws.tmc.ReadVReplicationWorkflow(ctx, target.GetPrimary().Tablet, &tabletmanagerdatapb.ReadVReplicationWorkflowRequest{
				Workflow: ts.WorkflowName(),
			})
			if err != nil {
				rec.RecordError(err)
				return nil
			}
			for _, stream := range res.Streams {
				if stream.Message != Frozen {
					rec.RecordError(fmt.Errorf("vreplication streams are not frozen on tablet %d", target.GetPrimary().Alias.Uid))
					return nil
				}
			}
			return nil
		})
	}
	wg.Wait()

	if !ts.keepRoutingRules {
		// Check if table is routable.
		if ts.MigrationType() == binlogdatapb.MigrationType_TABLES {
			rules, err := topotools.GetRoutingRules(ctx, ts.TopoServer())
			if err != nil {
				rec.RecordError(errors.New("could not get RoutingRules"))
			}
			for fromTable, toTables := range rules {
				for _, toTable := range toTables {
					for _, table := range ts.Tables() {

View on GitHub (pinned to 01a25a7d17)

Solutions

Wait until the streams are caught up and the freeze step marks them Frozen (check `_vt.vreplication` message on the target primary), then retry validation.
If a stream is stuck, investigate replication lag/errors on the target tablet before finalizing.
Ensure the freeze/finalize step of the workflow ran on all target primaries; re-run the workflow command that performs freezing.

Defensive patterns

Strategy: retry

Validate before calling

// confirm streams are frozen before finalizing
rows := queryTargetPrimary("select message from _vt.vreplication where workflow=?", wf)
for _, m := range rows {
    if m != "Frozen" {
        return fmt.Errorf("stream not frozen yet: %s", m)
    }
}

Try / catch

if err := validateWorkflowHasCompleted(ts); err != nil {
    if strings.Contains(err.Error(), "not frozen") {
        // streams still active: wait and retry validation
        time.Sleep(30 * time.Second)
        return retryValidate(ts)
    }
    return err
}

Prevention

Wait for streams to catch up and freeze before running Complete.
Monitor _vt.vreplication message state on target primaries.
Investigate stuck/lagged streams before retrying validation.

When it happens

Trigger: Running validateWorkflowHasCompleted (via validate/traffic finalization) while a target primary's vreplication streams still show a running/non-frozen state (message != "Frozen") — streams not yet caught up and frozen, or frozen step not executed.

Common situations: Calling Complete/StopTraffic too early while replication is still active; a target stream stuck (not catching up) so it never reaches Frozen; target primary unreachable so the query returns stale state.

Related errors

AI-assisted analysis of vitessio/vitess@01a25a7d17 (2026-09-01). Data as JSON: /api/errors/f9593d2b303ef3d5. Report an issue: GitHub.