{"record":{"id":"9ded14ad8ecc8110","repo":"vitessio/vitess","slug":"cannot-migrate-until-all-streams-are-running-s","errorCode":null,"errorMessage":"cannot migrate until all streams are running: %s: %d","messagePattern":"cannot migrate until all streams are running: (.+?): (.+?)","errorType":"console","errorClass":null,"httpStatus":null,"severity":"error","filePath":"go/vt/vtctl/workflow/stream_migrator.go","lineNumber":460,"sourceCode":"\t\tif !cancelMigrate {\n\t\t\t// This flow protects us from the following scenario: When we create streams,\n\t\t\t// we always do it in two phases. We start them off as Stopped, and then\n\t\t\t// update them to Running. If such an operation fails, we may be left with\n\t\t\t// lingering Stopped streams. They should actually be cleaned up by the user.\n\t\t\t// In the current workflow, we stop streams and restart them.\n\t\t\t// Once existing streams are stopped, there will be confusion about which of\n\t\t\t// them can be restarted because they will be no different from the lingering streams.\n\t\t\t// To prevent this confusion, we first check if there are any stopped streams.\n\t\t\t// If so, we request the operator to clean them up, or restart them before going ahead.\n\t\t\t// This allows us to assume that all stopped streams can be safely restarted\n\t\t\t// if we cancel the operation.\n\t\t\tstoppedStreams, err := sm.legacyReadTabletStreams(ctx, source.GetPrimary(), \"state = 'Stopped' and message != 'FROZEN'\")\n\t\t\tif err != nil {\n\t\t\t\treturn err\n\t\t\t}\n\n\t\t\tif len(stoppedStreams) != 0 {\n\t\t\t\treturn fmt.Errorf(\"cannot migrate until all streams are running: %s: %d\", source.GetShard().ShardName(), source.GetPrimary().Alias.Uid)\n\t\t\t}\n\t\t}\n\n\t\ttabletStreams, err := sm.legacyReadTabletStreams(ctx, source.GetPrimary(), \"\")\n\t\tif err != nil {\n\t\t\treturn err\n\t\t}\n\n\t\tif len(tabletStreams) == 0 {\n\t\t\t// No VReplication is running. So, we have no work to do.\n\t\t\treturn nil\n\t\t}\n\n\t\tquery := \"select distinct vrepl_id from _vt.copy_state where vrepl_id in \" + VReplicationStreams(tabletStreams).Values()\n\t\tp3qr, err := sm.ts.TabletManagerClient().VReplicationExec(ctx, source.GetPrimary().Tablet, query)\n\t\tswitch {\n\t\tcase err != nil:\n\t\t\treturn err","sourceCodeStart":442,"sourceCodeEnd":478,"githubUrl":"https://github.com/vitessio/vitess/blob/01a25a7d176f94613b8d59d799f438380a8760e4/go/vt/vtctl/workflow/stream_migrator.go#L442-L478","documentation":"During legacyReadSourceStreams (non-cancel path), the migrator first queries for streams in state 'Stopped' (message != 'FROZEN') on each source shard primary. Any non-frozen stopped stream is a hazard: after migration stops all streams, there would be no way to distinguish legitimately stopped streams from lingering half-created ones, so the migration aborts with the shard name and primary tablet alias UID.","triggerScenarios":"Calling MigrateStreams (or MoveTables/workflow stream migration) with cancelMigrate=false while at least one _vt.vreplication row on a source shard primary has state='Stopped' and is not FROZEN — typically a stream created in phase 1 (Stopped) that never got promoted to Running because a prior operation failed.","commonSituations":"A previous MoveTables/MigrateStreams attempt crashed between creating and starting streams; operator manually stopped a stream; copy phase failed leaving the stream Stopped; reshard left lingering Stopped streams the code comments say 'should actually be cleaned up by the user'.","solutions":["List stopped streams on the shard: `select id, workflow, message from _vt.vreplication where state='Stopped' and message != 'FROZEN'`.","Restart them: `update _vt.vreplication set state='Running' where id in (...)` or via vtctldclient, so they show Running before migrating.","If they are leftovers from a failed operation, delete them (`DELETE FROM _vt.vreplication WHERE id IN (...)`) and retry the migration.","Re-run the migration once no non-frozen Stopped streams remain on any source shard."],"exampleFix":"// before: migration fails due to lingering stopped stream id=3\n// after: clean up or restart it\nmysql> delete from _vt.vreplication where id=3 and state='Stopped';\n$ vtctldclient Workflow --keyspace customer Migrate ...","handlingStrategy":"validation","validationCode":"// before migrating, per source shard primary:\nqr, _ := tmclient.VReplicationExec(ctx, tablet, \"select id, workflow from _vt.vreplication where state='Stopped' and message != 'FROZEN'\")\nif len(qr.Rows) != 0 {\n    return fmt.Errorf(\"restart or delete %d stopped stream(s) before migrating\", len(qr.Rows))\n}","typeGuard":null,"tryCatchPattern":"err := migrateStreams(ctx, cfg)\nif err != nil && strings.Contains(err.Error(), \"cannot migrate until all streams are running\") {\n    // list Stopped non-frozen streams on the named shard, restart/delete them, retry\n}","preventionTips":["Clean up streams after failed workflow creations immediately","Avoid manually stopping streams on reshard source shards","Check state='Stopped' rows in _vt.vreplication as a pre-flight step"],"tags":["vitess","vreplication","workflow-migration","stream-state"],"backgroundTag":"stopped-vreplication-streams","analyzedSha":"01a25a7d176f94613b8d59d799f438380a8760e4","analyzedAt":"2026-09-01T17:28:30.605Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}