vitessio/vitess · error

cannot migrate until all streams are running: %s: %d

Error message

cannot migrate until all streams are running: %s: %d

What it means

During legacyReadSourceStreams (non-cancel path), the migrator first queries for streams in state 'Stopped' (message != 'FROZEN') on each source shard primary. Any non-frozen stopped stream is a hazard: after migration stops all streams, there would be no way to distinguish legitimately stopped streams from lingering half-created ones, so the migration aborts with the shard name and primary tablet alias UID.

Source

Thrown at go/vt/vtctl/workflow/stream_migrator.go:460

		if !cancelMigrate {
			// This flow protects us from the following scenario: When we create streams,
			// we always do it in two phases. We start them off as Stopped, and then
			// update them to Running. If such an operation fails, we may be left with
			// lingering Stopped streams. They should actually be cleaned up by the user.
			// In the current workflow, we stop streams and restart them.
			// Once existing streams are stopped, there will be confusion about which of
			// them can be restarted because they will be no different from the lingering streams.
			// To prevent this confusion, we first check if there are any stopped streams.
			// If so, we request the operator to clean them up, or restart them before going ahead.
			// This allows us to assume that all stopped streams can be safely restarted
			// if we cancel the operation.
			stoppedStreams, err := sm.legacyReadTabletStreams(ctx, source.GetPrimary(), "state = 'Stopped' and message != 'FROZEN'")
			if err != nil {
				return err
			}

			if len(stoppedStreams) != 0 {
				return fmt.Errorf("cannot migrate until all streams are running: %s: %d", source.GetShard().ShardName(), source.GetPrimary().Alias.Uid)
			}
		}

		tabletStreams, err := sm.legacyReadTabletStreams(ctx, source.GetPrimary(), "")
		if err != nil {
			return err
		}

		if len(tabletStreams) == 0 {
			// No VReplication is running. So, we have no work to do.
			return nil
		}

		query := "select distinct vrepl_id from _vt.copy_state where vrepl_id in " + VReplicationStreams(tabletStreams).Values()
		p3qr, err := sm.ts.TabletManagerClient().VReplicationExec(ctx, source.GetPrimary().Tablet, query)
		switch {
		case err != nil:
			return err

View on GitHub (pinned to 01a25a7d17)

Solutions

1. List stopped streams on the shard: `select id, workflow, message from _vt.vreplication where state='Stopped' and message != 'FROZEN'`.
2. Restart them: `update _vt.vreplication set state='Running' where id in (...)` or via vtctldclient, so they show Running before migrating.
3. If they are leftovers from a failed operation, delete them (`DELETE FROM _vt.vreplication WHERE id IN (...)`) and retry the migration.
4. Re-run the migration once no non-frozen Stopped streams remain on any source shard.

Example fix

// before: migration fails due to lingering stopped stream id=3
// after: clean up or restart it
mysql> delete from _vt.vreplication where id=3 and state='Stopped';
$ vtctldclient Workflow --keyspace customer Migrate ...

// before migrating, per source shard primary:
qr, _ := tmclient.VReplicationExec(ctx, tablet, "select id, workflow from _vt.vreplication where state='Stopped' and message != 'FROZEN'")
if len(qr.Rows) != 0 {
    return fmt.Errorf("restart or delete %d stopped stream(s) before migrating", len(qr.Rows))
}

err := migrateStreams(ctx, cfg)
if err != nil && strings.Contains(err.Error(), "cannot migrate until all streams are running") {
    // list Stopped non-frozen streams on the named shard, restart/delete them, retry
}

When it happens

Trigger: Calling MigrateStreams (or MoveTables/workflow stream migration) with cancelMigrate=false while at least one _vt.vreplication row on a source shard primary has state='Stopped' and is not FROZEN — typically a stream created in phase 1 (Stopped) that never got promoted to Running because a prior operation failed.

Common situations: A previous MoveTables/MigrateStreams attempt crashed between creating and starting streams; operator manually stopped a stream; copy phase failed leaving the stream Stopped; reshard left lingering Stopped streams the code comments say 'should actually be cleaned up by the user'.

Related errors

• VReplication stream has the same workflow name as the reshar
• VReplication streams must have named workflows for migration
• VReplication stream has the same workflow name as the reshar
• cannot migrate while vreplication streams in source shards a
• streams are mismatched across source shards: %s vs %s

AI-assisted analysis of vitessio/vitess@01a25a7d17 (2026-09-01). Data as JSON: /api/errors/9ded14ad8ecc8110. Report an issue: GitHub.