vitessio/vitess · error

could not restart source streams: %v

Error message

could not restart source streams: %v

What it means

The second step of CancelStreamMigrations restarts the source streams by running an UPDATE on _vt.vreplication setting state='Running' on every source primary, excluding the workflow's reverse streams. If this VReplicationExec fails on any source, the error is recorded and aggregated with any earlier cleanup errors.

Source

Thrown at go/vt/vtctl/workflow/stream_migrator.go:226

	}
	errs := &concurrency.AllErrorRecorder{}

	if err := sm.deleteTargetStreams(ctx); err != nil {
		errs.RecordError(fmt.Errorf("could not delete target streams: %v", err))
	}

	// Restart the source streams, but leave the Reshard workflow's reverse
	// variant stopped.
	err := sm.ts.ForAllSources(func(source *MigrationSource) error {
		// We intend to update all but our workflow's reverse streams, so we
		// indicate that it's safe in this case using the comment diretive.
		query := fmt.Sprintf("update /*vt+ %s */ _vt.vreplication set state='Running', stop_pos=null, message='' where db_name=%s and workflow != %s",
			vreplication.AllowUnsafeWriteCommentDirective, encodeString(source.GetPrimary().DbName()), encodeString(sm.ts.ReverseWorkflowName()))
		_, err := sm.ts.VReplicationExec(ctx, source.GetPrimary().Alias, query)
		return err
	})
	if err != nil {
		errs.RecordError(fmt.Errorf("could not restart source streams: %v", err))
		sm.logger.Errorf("Cancel stream migrations failed: could not restart source streams: %v", err)
	}
	if errs.HasErrors() {
		return errs.AggrError(vterrors.Aggregate)
	}
	return nil
}

// MigrateStreams migrates N streams
func (sm *StreamMigrator) MigrateStreams(ctx context.Context) error {
	if sm.streams == nil {
		return nil
	}

	if err := sm.deleteTargetStreams(ctx); err != nil {
		return err
	}

View on GitHub (pinned to 01a25a7d17)

Solutions

1. Verify each source shard has a healthy, reachable primary and retry the cancel.
2. Check tablet logs for the underlying VReplicationExec failure (the aggregated error includes it via %v).
3. If needed, run the equivalent UPDATE on the affected source tablets manually, then re-run cancel.
4. Complete any in-progress tablet restarts/maintenance before cancelling.

// Check source primaries accept writes before cancelling
for _, shard := range sourceShards {
    if !tabletHealthy(shard.Primary) { return fmt.Errorf("source primary %v unhealthy", shard) }
}

err := client.CancelStreamMigration(ctx, keyspace, workflow)
if err != nil && strings.Contains(err.Error(), "could not restart source streams") {
    // inspect tablet logs for the VReplicationExec failure, fix, retry
}

When it happens

Trigger: CancelStreamMigration while a source shard primary is unreachable, rejects the update, or the SQL execution on the tablet fails; the reverse workflow name excluded in the query does not match rows as expected (that only affects which rows restart, not the error).

Common situations: Source primary tablet down or restarted mid-cancel; permissions/db_name mismatch causing zero-effect or failing updates; network issues between control plane and source tablets during a rolling maintenance.

Related errors

• could not delete target streams: %v
• ErrCannotUpdateImmutableColumn
• health check failed on %s
• value out of range
• both atomic copy and partial mode cannot be specified for th

AI-assisted analysis of vitessio/vitess@01a25a7d17 (2026-09-01). Data as JSON: /api/errors/1304a87b8fc5ece1. Report an issue: GitHub.