{"record":{"id":"06433fca10ab3b99","repo":"vitessio/vitess","slug":"s-stream-d-position-s-does-not-match-s","errorCode":null,"errorMessage":"%s: stream %d position: %s does not match %s","messagePattern":"(.+?): stream (.+?) position: (.+?) does not match (.+?)","errorType":"console","errorClass":null,"httpStatus":null,"severity":"error","filePath":"go/vt/vtctl/workflow/stream_migrator.go","lineNumber":898,"sourceCode":"\n\t// This is not really required because it's not used later.\n\t// But we keep it up-to-date for good measure.\n\tsm.streams = stoppedStreams\n\n\tvar (\n\t\toneSet    []*VReplicationStream\n\t\tallErrors concurrency.AllErrorRecorder\n\t)\n\n\tfor _, tabletStreams := range stoppedStreams {\n\t\tif oneSet == nil {\n\t\t\toneSet = tabletStreams\n\t\t}\n\n\t\tfor _, vrs := range tabletStreams {\n\t\t\tkey := fmt.Sprintf(\"%s:%s\", vrs.BinlogSource.Keyspace, vrs.BinlogSource.Shard)\n\t\t\tif pos := stopPositions[key]; !vrs.Position.Equal(pos) {\n\t\t\t\tallErrors.RecordError(fmt.Errorf(\"%s: stream %d position: %s does not match %s\", key, vrs.ID, replication.EncodePosition(vrs.Position), replication.EncodePosition(pos)))\n\t\t\t}\n\t\t}\n\t}\n\n\tif allErrors.HasErrors() {\n\t\treturn nil, allErrors.AggrError(vterrors.Aggregate)\n\t}\n\n\tsm.templates, err = sm.templatize(ctx, oneSet)\n\tif err != nil {\n\t\t// Unreachable: we've already templatized this before.\n\t\treturn nil, err\n\t}\n\n\treturn VReplicationStreams(sm.templates).Workflows(), allErrors.AggrError(vterrors.Aggregate)\n}\n\nfunc (sm *StreamMigrator) verifyStreamPositions(ctx context.Context, stopPositions map[string]replication.Position) ([]string, error) {","sourceCodeStart":880,"sourceCodeEnd":916,"githubUrl":"https://github.com/vitessio/vitess/blob/01a25a7d176f94613b8d59d799f438380a8760e4/go/vt/vtctl/workflow/stream_migrator.go#L880-L916","documentation":"legacyVerifyStreamPositions, called by LegacyStopStreams, re-reads the stopped streams and compares each stream's recorded position against the expected stop positions (keyed by \"keyspace:shard\") that were computed earlier. A mismatch means the stream's position changed or the stop-position map is keyed/stale differently, so the workflow migration cannot safely proceed — the streams may not have stopped where expected.","triggerScenarios":"Calling LegacyStopStreams where, for a stream on a source shard, vrs.Position (read after stopping) does not equal stopPositions[keyspace:shard]. Happens if writes continued past the stop point, streams were restarted/repositioned between capture and verify, or the keyspace/shard key is absent from the map (zero position) so nothing matches.","commonSituations":"Race where another actor (or legacy vs modern tooling) restarted the stream; manual GTID manipulation or emergency reparent between capture and verify; stale stopPositions map because shard naming changed (e.g. reshard in progress); using the legacy wrangler path with rows written by newer tooling.","solutions":["Check the stream state and message: `select id, state, position, message from _vt.vreplication` on the source primary to see if the stream moved after stopping.","Re-run the migration/stop sequence cleanly (cancel target streams and restart the MoveTables/Reshard migration stop) so positions are recaptured and verified atomically.","Confirm no other process (cron, monitoring, old vtctld) is touching _vt.vreplication during the cutover window.","If on the legacy code path, switch to the current StopStreams path (newer vtctldclient) which handles position verification more robustly."],"exampleFix":"// before: stopPositions captured at t0, stream repositioned at t1 -> verify fails\n// after: cancel and redo the cutover so capture+verify are one operation\nvtctldclient MoveTables --target-keyspace ks Complete   // or restart MigrateStreams","handlingStrategy":"retry","validationCode":"// Ensure streams are Stopped and no writers touch _vt.vreplication during cutover\n-- SELECT id, state FROM _vt.vreplication;  -- all must be 'Stopped' before verify","typeGuard":null,"tryCatchPattern":"// On mismatch, redo the cutover atomically\nif err := wr.LegacyStopStreams(ctx, ...); err != nil {\n    // positions drifted: cancel target streams and restart the cutover from scratch\n    _ = wr.DeleteTargetStreams(ctx)\n    return retryCutover(ctx)\n}","preventionTips":["Block application writes before capturing stop positions.","No concurrent admin/cron jobs may modify vreplication during cutover.","Prefer the modern StopStreams path over legacy wrangler code."],"tags":["vitess","vreplication","gtid-position","race-condition","legacy-path"],"backgroundTag":"vreplication-position-mismatch","analyzedSha":"01a25a7d176f94613b8d59d799f438380a8760e4","analyzedAt":"2026-09-01T17:28:30.605Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}