{"record":{"id":"498e098ce6c0fb22","repo":"vitessio/vitess","slug":"refreshstate-v-failed-to-acquire-toporeadpool","errorCode":null,"errorMessage":"RefreshState(%v) failed to acquire topoReadPool: %w","messagePattern":"RefreshState\\((.+?)\\) failed to acquire topoReadPool: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"go/vt/vtadmin/cluster/cluster.go","lineNumber":2124,"sourceCode":"\treturn &vtadminpb.PlannedFailoverShardResponse{\n\t\tCluster:         c.ToProto(),\n\t\tKeyspace:        resp.Keyspace,\n\t\tShard:           resp.Shard,\n\t\tPromotedPrimary: resp.PromotedPrimary,\n\t\tEvents:          resp.Events,\n\t}, nil\n}\n\n// RefreshState reloads the tablet record from a cluster's topo on a tablet.\nfunc (c *Cluster) RefreshState(ctx context.Context, tablet *vtadminpb.Tablet) error {\n\tspan, ctx := trace.NewSpan(ctx, \"Cluster.RefreshState\")\n\tdefer span.Finish()\n\n\tAnnotateSpan(c, span)\n\tspan.Annotate(\"tablet_alias\", topoproto.TabletAliasString(tablet.Tablet.Alias))\n\n\tif err := c.topoReadPool.Acquire(ctx); err != nil {\n\t\treturn fmt.Errorf(\"RefreshState(%v) failed to acquire topoReadPool: %w\", topoproto.TabletAliasString(tablet.Tablet.Alias), err)\n\t}\n\tdefer c.topoReadPool.Release()\n\n\t_, err := c.Vtctld.RefreshState(ctx, &vtctldatapb.RefreshStateRequest{\n\t\tTabletAlias: tablet.Tablet.Alias,\n\t})\n\treturn err\n}\n\n// RefreshTabletReplicationSource performs a `CHANGE REPLICATION SOURCE TO` on\n// a tablet to replicate from the current primary in the shard.\nfunc (c *Cluster) RefreshTabletReplicationSource(ctx context.Context, tablet *vtadminpb.Tablet) (*vtadminpb.RefreshTabletReplicationSourceResponse, error) {\n\tspan, ctx := trace.NewSpan(ctx, \"Cluster.RefreshTabletReplicationSource\")\n\tdefer span.Finish()\n\n\tAnnotateSpan(c, span)\n\tspan.Annotate(\"tablet_alias\", topoproto.TabletAliasString(tablet.Tablet.Alias))\n","sourceCodeStart":2106,"sourceCodeEnd":2142,"githubUrl":"https://github.com/vitessio/vitess/blob/01a25a7d176f94613b8d59d799f438380a8760e4/go/vt/vtadmin/cluster/cluster.go#L2106-L2142","documentation":"Cluster.RefreshState serializes read-only topo operations through topoReadPool. When Acquire fails — because the context is cancelled or the pool slot limit is reached — the error is wrapped with the tablet alias and returned.","triggerScenarios":"Calling Cluster.RefreshState (vtadmin RefreshState RPC) while topoReadPool is saturated with other topo reads, or after the request context has been cancelled/timed out.","commonSituations":"Bulk tablet refresh loops over many tablets exhausting the read pool; dashboards polling RefreshState concurrently; slow topo backend (etcd latency) making waiters time out.","solutions":["Retry the RefreshState call with a fresh context","Throttle or batch client-side refresh loops to stay within pool capacity","Increase topo_read_pool_size in the vtadmin cluster config","Check topo backend (etcd/zk) health and latency"],"exampleFix":"// before\nfor _, t := range tablets {\n    go c.RefreshState(ctx, t) // floods topoReadPool\n}\n// after\nsem := make(chan struct{}, 2)\nfor _, t := range tablets {\n    sem <- struct{}{}\n    go func(t *vtadminpb.Tablet) { defer func() { <-sem }(); c.RefreshState(ctx, t) }(t)\n}","handlingStrategy":"retry","validationCode":"if ctx.Err() != nil {\n    return ctx.Err()\n}","typeGuard":null,"tryCatchPattern":"err := c.RefreshState(ctx, tablet)\nif err != nil && strings.Contains(err.Error(), \"failed to acquire topoReadPool\") {\n    time.Sleep(backoff)\n    err = c.RefreshState(ctx, tablet)\n}","preventionTips":["Throttle concurrent RefreshState calls client-side","Monitor topo backend (etcd/zk) latency","Raise topo_read_pool_size if bulk refreshes are routine"],"tags":["vtadmin","concurrency","topo","context-cancelled"],"backgroundTag":"semaphore-acquisition-failed","analyzedSha":"01a25a7d176f94613b8d59d799f438380a8760e4","analyzedAt":"2026-09-01T17:28:30.605Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}