vitessio/vitess · error

could not restart source streams

Error message

could not restart source streams: %v

What it means

The second step of CancelStreamMigrations restarts the source streams by running an UPDATE on _vt.vreplication setting state='Running' on every source primary, excluding the workflow's reverse streams. If this VReplicationExec fails on any source, the error is recorded and aggregated with any earlier cleanup errors.

Solutions

1. Verify each source shard has a healthy, reachable primary and retry the cancel.
2. Check tablet logs for the underlying VReplicationExec failure (the aggregated error includes it via %v).
3. If needed, run the equivalent UPDATE on the affected source tablets manually, then re-run cancel.
4. Complete any in-progress tablet restarts/maintenance before cancelling.

// Check source primaries accept writes before cancelling
for _, shard := range sourceShards {
    if !tabletHealthy(shard.Primary) { return fmt.Errorf("source primary %v unhealthy", shard) }
}

err := client.CancelStreamMigration(ctx, keyspace, workflow)
if err != nil && strings.Contains(err.Error(), "could not restart source streams") {
    // inspect tablet logs for the VReplicationExec failure, fix, retry
}

When it happens

Trigger: CancelStreamMigration while a source shard primary is unreachable, rejects the update, or the SQL execution on the tablet fails; the reverse workflow name excluded in the query does not match rows as expected (that only affects which rows restart, not the error).

Common situations: Source primary tablet down or restarted mid-cancel; permissions/db_name mismatch causing zero-effect or failing updates; network issues between control plane and source tablets during a rolling maintenance.

Related errors

• could not delete target streams
• ErrCannotUpdateImmutableColumn
• health check failed on
• a backup is already running on tablet
• attempted to execute unprepared query

AI-assisted analysis of vitessio/vitess@01a25a7d17 (2026-09-01). Data as JSON: /api/errors/1304a87b8fc5ece1. Report an issue: GitHub.

Appendix: source

Thrown at go/vt/vtctl/workflow/stream_migrator.go:226

	}
	errs := &concurrency.AllErrorRecorder{}

	if err := sm.deleteTargetStreams(ctx); err != nil {
		errs.RecordError(fmt.Errorf("could not delete target streams: %v", err))
	}

	// Restart the source streams, but leave the Reshard workflow's reverse
	// variant stopped.
	err := sm.ts.ForAllSources(func(source *MigrationSource) error {
		// We intend to update all but our workflow's reverse streams, so we
		// indicate that it's safe in this case using the comment diretive.
		query := fmt.Sprintf("update /*vt+ %s */ _vt.vreplication set state='Running', stop_pos=null, message='' where db_name=%s and workflow != %s",
			vreplication.AllowUnsafeWriteCommentDirective, encodeString(source.GetPrimary().DbName()), encodeString(sm.ts.ReverseWorkflowName()))
		_, err := sm.ts.VReplicationExec(ctx, source.GetPrimary().Alias, query)
		return err
	})
	if err != nil {
		errs.RecordError(fmt.Errorf("could not restart source streams: %v", err))
		sm.logger.Errorf("Cancel stream migrations failed: could not restart source streams: %v", err)
	}
	if errs.HasErrors() {
		return errs.AggrError(vterrors.Aggregate)
	}
	return nil
}

// MigrateStreams migrates N streams
func (sm *StreamMigrator) MigrateStreams(ctx context.Context) error {
	if sm.streams == nil {
		return nil
	}

	if err := sm.deleteTargetStreams(ctx); err != nil {
		return err
	}

View on GitHub (pinned to 01a25a7d17)