vitessio/vitess · error

%s: stream %d position: %s does not match %s

Error message

%s: stream %d position: %s does not match %s

What it means

legacyVerifyStreamPositions, called by LegacyStopStreams, re-reads the stopped streams and compares each stream's recorded position against the expected stop positions (keyed by "keyspace:shard") that were computed earlier. A mismatch means the stream's position changed or the stop-position map is keyed/stale differently, so the workflow migration cannot safely proceed — the streams may not have stopped where expected.

Source

Thrown at go/vt/vtctl/workflow/stream_migrator.go:898

	// This is not really required because it's not used later.
	// But we keep it up-to-date for good measure.
	sm.streams = stoppedStreams

	var (
		oneSet    []*VReplicationStream
		allErrors concurrency.AllErrorRecorder
	)

	for _, tabletStreams := range stoppedStreams {
		if oneSet == nil {
			oneSet = tabletStreams
		}

		for _, vrs := range tabletStreams {
			key := fmt.Sprintf("%s:%s", vrs.BinlogSource.Keyspace, vrs.BinlogSource.Shard)
			if pos := stopPositions[key]; !vrs.Position.Equal(pos) {
				allErrors.RecordError(fmt.Errorf("%s: stream %d position: %s does not match %s", key, vrs.ID, replication.EncodePosition(vrs.Position), replication.EncodePosition(pos)))
			}
		}
	}

	if allErrors.HasErrors() {
		return nil, allErrors.AggrError(vterrors.Aggregate)
	}

	sm.templates, err = sm.templatize(ctx, oneSet)
	if err != nil {
		// Unreachable: we've already templatized this before.
		return nil, err
	}

	return VReplicationStreams(sm.templates).Workflows(), allErrors.AggrError(vterrors.Aggregate)
}

func (sm *StreamMigrator) verifyStreamPositions(ctx context.Context, stopPositions map[string]replication.Position) ([]string, error) {

View on GitHub (pinned to 01a25a7d17)

Solutions

1. Check the stream state and message: `select id, state, position, message from _vt.vreplication` on the source primary to see if the stream moved after stopping.
2. Re-run the migration/stop sequence cleanly (cancel target streams and restart the MoveTables/Reshard migration stop) so positions are recaptured and verified atomically.
3. Confirm no other process (cron, monitoring, old vtctld) is touching _vt.vreplication during the cutover window.
4. If on the legacy code path, switch to the current StopStreams path (newer vtctldclient) which handles position verification more robustly.

Example fix

// before: stopPositions captured at t0, stream repositioned at t1 -> verify fails
// after: cancel and redo the cutover so capture+verify are one operation
vtctldclient MoveTables --target-keyspace ks Complete   // or restart MigrateStreams

// Ensure streams are Stopped and no writers touch _vt.vreplication during cutover
-- SELECT id, state FROM _vt.vreplication;  -- all must be 'Stopped' before verify

// On mismatch, redo the cutover atomically
if err := wr.LegacyStopStreams(ctx, ...); err != nil {
    // positions drifted: cancel target streams and restart the cutover from scratch
    _ = wr.DeleteTargetStreams(ctx)
    return retryCutover(ctx)
}

When it happens

Trigger: Calling LegacyStopStreams where, for a stream on a source shard, vrs.Position (read after stopping) does not equal stopPositions[keyspace:shard]. Happens if writes continued past the stop point, streams were restarted/repositioned between capture and verify, or the keyspace/shard key is absent from the map (zero position) so nothing matches.

Common situations: Race where another actor (or legacy vs modern tooling) restarted the stream; manual GTID manipulation or emergency reparent between capture and verify; stale stopPositions map because shard naming changed (e.g. reshard in progress); using the legacy wrangler path with rows written by newer tooling.

Related errors

• both atomic copy and partial mode cannot be specified for th
• invalid workflow
• multiple source keyspaces for a single workflow
• multiple target keyspaces for a single workflow
• no streams found

AI-assisted analysis of vitessio/vitess@01a25a7d17 (2026-09-01). Data as JSON: /api/errors/06433fca10ab3b99. Report an issue: GitHub.