{"record":{"id":"44ec34ba418dca72","repo":"vitessio/vitess","slug":"failed-to-successfully-refresh-all-tablets-in-the","errorCode":null,"errorMessage":"failed to successfully refresh all tablets in the %s/%s source shard (%v):\n  %v","messagePattern":"failed to successfully refresh all tablets in the (.+?)/(.+?) source shard \\((.+?)\\):\n  (.+?)","errorType":"console","errorClass":null,"httpStatus":null,"severity":"error","filePath":"go/vt/vtctl/workflow/traffic_switcher.go","lineNumber":500,"sourceCode":"\t\t\treturn si.UpdateDeniedTables(ctx, topo.UpdateDeniedTablesOpts{\n\t\t\t\tRemove:     true,\n\t\t\t\tTables:     ts.Tables(),\n\t\t\t\tTabletType: topodatapb.TabletType_PRIMARY,\n\t\t\t})\n\t\t}); err != nil {\n\t\t\treturn err\n\t\t}\n\t\trtbsCtx, cancel := context.WithTimeout(ctx, shardTabletRefreshTimeout)\n\t\tdefer cancel()\n\t\tisPartial, partialDetails, err := topotools.RefreshTabletsByShard(rtbsCtx, ts.TopoServer(), ts.TabletManagerClient(), source.GetShard(), nil, ts.Logger())\n\t\tif isPartial {\n\t\t\tmsg := fmt.Sprintf(\"failed to successfully refresh all tablets in the %s/%s source shard (%v):\\n  %v\",\n\t\t\t\tsource.GetShard().Keyspace(), source.GetShard().ShardName(), err, partialDetails)\n\t\t\tif ts.force {\n\t\t\t\tlog.Warn(msg)\n\t\t\t\treturn nil\n\t\t\t} else {\n\t\t\t\treturn errors.New(msg)\n\t\t\t}\n\t\t}\n\t\treturn err\n\t})\n}\n\nfunc (ts *trafficSwitcher) dropTargetDeniedTables(ctx context.Context) error {\n\treturn ts.ForAllTargets(func(target *MigrationTarget) error {\n\t\tif _, err := ts.TopoServer().UpdateShardFields(ctx, ts.TargetKeyspaceName(), target.GetShard().ShardName(), func(si *topo.ShardInfo) error {\n\t\t\treturn si.UpdateDeniedTables(ctx, topo.UpdateDeniedTablesOpts{\n\t\t\t\tRemove:     true,\n\t\t\t\tTables:     ts.Tables(),\n\t\t\t\tTabletType: topodatapb.TabletType_PRIMARY,\n\t\t\t})\n\t\t}); err != nil {\n\t\t\treturn err\n\t\t}\n\t\trtbsCtx, cancel := context.WithTimeout(ctx, shardTabletRefreshTimeout)","sourceCodeStart":482,"sourceCodeEnd":518,"githubUrl":"https://github.com/vitessio/vitess/blob/01a25a7d176f94613b8d59d799f438380a8760e4/go/vt/vtctl/workflow/traffic_switcher.go#L482-L518","documentation":"After switching traffic, the source shard's tablets must all be refreshed (reloaded routing/permissions). If any source tablet fails to refresh, this error lists the keyspace, shard, underlying error and the partial per-tablet details. With ts.force set the failure is only logged as a warning and the switch proceeds; otherwise the operation fails.","triggerScenarios":"Running SwitchReads/SwitchWrites where one or more tablets in the source shard (source.GetShard()) return errors during the refresh RPC, with partialDetails describing which tablets failed.","commonSituations":"A source tablet is down, slow, or unreachable (network partition, tablet restarting) during a MoveTables/Reshard cutover.","solutions":["Fix the failing source tablets (restart vttablet, restore connectivity) and retry the traffic switch.","Use the force option only if the partial refresh is acceptable, acknowledging stale routing on the reported tablets.","Check vttablet logs on the listed tablets for the underlying refresh error and address it directly."],"exampleFix":"// before\nreturn errors.New(msg) // switch aborted\n// after\n// pass --force to the switch command if refresh failure is acceptable:\nvtctldclient SwitchReads --tablet_type=replica --force commerce.sales","handlingStrategy":"retry","validationCode":"for _, t := range sourceShardTablets {\n    if !tabletHealthy(ctx, t) { return fmt.Errorf(\"tablet %s unhealthy before switch\", t) }\n}","typeGuard":null,"tryCatchPattern":"err := switchTraffic(ctx, ks, wf)\nif isTabletRefreshErr(err) && isRetryable(err) {\n    err = retry.Do(ctx, 3, func() error { return switchTraffic(ctx, ks, wf) })\n}","preventionTips":["Verify all source tablets are serving before a cutover","Use --force only when stale routing is acceptable","Watch tablet health during the maintenance window"],"tags":["vitess","traffic-switch","tablet-refresh","source-shard"],"backgroundTag":"tablet-refresh-failed","analyzedSha":"01a25a7d176f94613b8d59d799f438380a8760e4","analyzedAt":"2026-09-01T17:28:30.605Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}