What causes "could not delete target streams: %v" in vitessio/vitess?

Check target shard primary tablet health and retry CancelStreamMigration.

could not delete target streams: %v

Error message

could not delete target streams: %v

What it means

CancelStreamMigrations rolls back a stream migration. The first rollback step deletes the VReplication streams that were created on the target shards. If deleteTargetStreams fails, the error is wrapped and recorded, and cancellation continues with restarting source streams; all collected errors are aggregated and returned at the end.

Source

Thrown at go/vt/vtctl/workflow/stream_migrator.go:212

	}

	return streams
}

// Templates returns a copy of the StreamMigrator's template streams.
func (sm *StreamMigrator) Templates() []*VReplicationStream {
	return VReplicationStreams(sm.templates).Copy().ToSlice()
}

// CancelStreamMigrations cancels the stream migrations.
func (sm *StreamMigrator) CancelStreamMigrations(ctx context.Context) error {
	if sm.streams == nil {
		return nil
	}
	errs := &concurrency.AllErrorRecorder{}

	if err := sm.deleteTargetStreams(ctx); err != nil {
		errs.RecordError(fmt.Errorf("could not delete target streams: %v", err))
	}

	// Restart the source streams, but leave the Reshard workflow's reverse
	// variant stopped.
	err := sm.ts.ForAllSources(func(source *MigrationSource) error {
		// We intend to update all but our workflow's reverse streams, so we
		// indicate that it's safe in this case using the comment diretive.
		query := fmt.Sprintf("update /*vt+ %s */ _vt.vreplication set state='Running', stop_pos=null, message='' where db_name=%s and workflow != %s",
			vreplication.AllowUnsafeWriteCommentDirective, encodeString(source.GetPrimary().DbName()), encodeString(sm.ts.ReverseWorkflowName()))
		_, err := sm.ts.VReplicationExec(ctx, source.GetPrimary().Alias, query)
		return err
	})
	if err != nil {
		errs.RecordError(fmt.Errorf("could not restart source streams: %v", err))
		sm.logger.Errorf("Cancel stream migrations failed: could not restart source streams: %v", err)
	}
	if errs.HasErrors() {
		return errs.AggrError(vterrors.Aggregate)

View on GitHub (pinned to 01a25a7d17)

Solutions

Check target shard primary tablet health and retry CancelStreamMigration.
Fix tablet connectivity (vtctld to tablet gRPC) and re-run the cancel.
Manually delete the leftover _vt.vreplication rows on the target shards if automated cancel keeps failing.
Re-run the full cancel flow; it is idempotent for already-deleted streams.

Defensive patterns

Strategy: retry

Validate before calling

// Check target primaries are reachable before cancelling
for _, shard := range targetShards {
    if !tabletHealthy(shard.Primary) { return fmt.Errorf("target primary %v unhealthy", shard) }
}

Try / catch

err := client.CancelStreamMigration(ctx, keyspace, workflow)
if err != nil && strings.Contains(err.Error(), "could not delete target streams") {
    // fix target tablet health, then retry the cancel (idempotent)
}

Prevention

Confirm all target shard primaries are healthy before cancelling
Monitor aggregated error output for the underlying per-shard cause
Clean leftover target streams after repeated failed cancels

When it happens

Trigger: Calling CancelStreamMigration (via vtctld/workflow API) when target-shard vreplication deletion fails — e.g. a target primary tablet is unreachable, VReplicationExec errors, or the target shards have no healthy primary.

Common situations: Target primary tablet down or in a bad state during cancel; network partition between vtctld and target tablets; target keyspace already partially cleaned by a previous failed attempt.

Related errors

AI-assisted analysis of vitessio/vitess@01a25a7d17 (2026-09-01). Data as JSON: /api/errors/c96cba7d3f3de607. Report an issue: GitHub.