failed to successfully refresh all tablets in the %s/%s sour

Error message

failed to successfully refresh all tablets in the %s/%s source shard (%v):
  %v

What it means

After switching traffic, the source shard's tablets must all be refreshed (reloaded routing/permissions). If any source tablet fails to refresh, this error lists the keyspace, shard, underlying error and the partial per-tablet details. With ts.force set the failure is only logged as a warning and the switch proceeds; otherwise the operation fails.

Source

Thrown at go/vt/vtctl/workflow/traffic_switcher.go:500

			return si.UpdateDeniedTables(ctx, topo.UpdateDeniedTablesOpts{
				Remove:     true,
				Tables:     ts.Tables(),
				TabletType: topodatapb.TabletType_PRIMARY,
			})
		}); err != nil {
			return err
		}
		rtbsCtx, cancel := context.WithTimeout(ctx, shardTabletRefreshTimeout)
		defer cancel()
		isPartial, partialDetails, err := topotools.RefreshTabletsByShard(rtbsCtx, ts.TopoServer(), ts.TabletManagerClient(), source.GetShard(), nil, ts.Logger())
		if isPartial {
			msg := fmt.Sprintf("failed to successfully refresh all tablets in the %s/%s source shard (%v):\n  %v",
				source.GetShard().Keyspace(), source.GetShard().ShardName(), err, partialDetails)
			if ts.force {
				log.Warn(msg)
				return nil
			} else {
				return errors.New(msg)
			}
		}
		return err
	})
}

func (ts *trafficSwitcher) dropTargetDeniedTables(ctx context.Context) error {
	return ts.ForAllTargets(func(target *MigrationTarget) error {
		if _, err := ts.TopoServer().UpdateShardFields(ctx, ts.TargetKeyspaceName(), target.GetShard().ShardName(), func(si *topo.ShardInfo) error {
			return si.UpdateDeniedTables(ctx, topo.UpdateDeniedTablesOpts{
				Remove:     true,
				Tables:     ts.Tables(),
				TabletType: topodatapb.TabletType_PRIMARY,
			})
		}); err != nil {
			return err
		}
		rtbsCtx, cancel := context.WithTimeout(ctx, shardTabletRefreshTimeout)

View on GitHub (pinned to 01a25a7d17)

Solutions

Fix the failing source tablets (restart vttablet, restore connectivity) and retry the traffic switch.
Use the force option only if the partial refresh is acceptable, acknowledging stale routing on the reported tablets.
Check vttablet logs on the listed tablets for the underlying refresh error and address it directly.

Example fix

// before
return errors.New(msg) // switch aborted
// after
// pass --force to the switch command if refresh failure is acceptable:
vtctldclient SwitchReads --tablet_type=replica --force commerce.sales

Defensive patterns

Strategy: retry

Validate before calling

for _, t := range sourceShardTablets {
    if !tabletHealthy(ctx, t) { return fmt.Errorf("tablet %s unhealthy before switch", t) }
}

Try / catch

err := switchTraffic(ctx, ks, wf)
if isTabletRefreshErr(err) && isRetryable(err) {
    err = retry.Do(ctx, 3, func() error { return switchTraffic(ctx, ks, wf) })
}

Prevention

Verify all source tablets are serving before a cutover
Use --force only when stale routing is acceptable
Watch tablet health during the maintenance window

When it happens

Trigger: Running SwitchReads/SwitchWrites where one or more tablets in the source shard (source.GetShard()) return errors during the refresh RPC, with partialDetails describing which tablets failed.

Common situations: A source tablet is down, slow, or unreachable (network partition, tablet restarting) during a MoveTables/Reshard cutover.

Related errors

AI-assisted analysis of vitessio/vitess@01a25a7d17 (2026-09-01). Data as JSON: /api/errors/44ec34ba418dca72. Report an issue: GitHub.