vitessio/vitess · error

RefreshState(%v) failed to acquire topoReadPool: %w

Error message

RefreshState(%v) failed to acquire topoReadPool: %w

What it means

Cluster.RefreshState serializes read-only topo operations through topoReadPool. When Acquire fails — because the context is cancelled or the pool slot limit is reached — the error is wrapped with the tablet alias and returned.

Source

Thrown at go/vt/vtadmin/cluster/cluster.go:2124

	return &vtadminpb.PlannedFailoverShardResponse{
		Cluster:         c.ToProto(),
		Keyspace:        resp.Keyspace,
		Shard:           resp.Shard,
		PromotedPrimary: resp.PromotedPrimary,
		Events:          resp.Events,
	}, nil
}

// RefreshState reloads the tablet record from a cluster's topo on a tablet.
func (c *Cluster) RefreshState(ctx context.Context, tablet *vtadminpb.Tablet) error {
	span, ctx := trace.NewSpan(ctx, "Cluster.RefreshState")
	defer span.Finish()

	AnnotateSpan(c, span)
	span.Annotate("tablet_alias", topoproto.TabletAliasString(tablet.Tablet.Alias))

	if err := c.topoReadPool.Acquire(ctx); err != nil {
		return fmt.Errorf("RefreshState(%v) failed to acquire topoReadPool: %w", topoproto.TabletAliasString(tablet.Tablet.Alias), err)
	}
	defer c.topoReadPool.Release()

	_, err := c.Vtctld.RefreshState(ctx, &vtctldatapb.RefreshStateRequest{
		TabletAlias: tablet.Tablet.Alias,
	})
	return err
}

// RefreshTabletReplicationSource performs a `CHANGE REPLICATION SOURCE TO` on
// a tablet to replicate from the current primary in the shard.
func (c *Cluster) RefreshTabletReplicationSource(ctx context.Context, tablet *vtadminpb.Tablet) (*vtadminpb.RefreshTabletReplicationSourceResponse, error) {
	span, ctx := trace.NewSpan(ctx, "Cluster.RefreshTabletReplicationSource")
	defer span.Finish()

	AnnotateSpan(c, span)
	span.Annotate("tablet_alias", topoproto.TabletAliasString(tablet.Tablet.Alias))

View on GitHub (pinned to 01a25a7d17)

Solutions

1. Retry the RefreshState call with a fresh context
2. Throttle or batch client-side refresh loops to stay within pool capacity
3. Increase topo_read_pool_size in the vtadmin cluster config
4. Check topo backend (etcd/zk) health and latency

Example fix

// before
for _, t := range tablets {
    go c.RefreshState(ctx, t) // floods topoReadPool
}
// after
sem := make(chan struct{}, 2)
for _, t := range tablets {
    sem <- struct{}{}
    go func(t *vtadminpb.Tablet) { defer func() { <-sem }(); c.RefreshState(ctx, t) }(t)
}

if ctx.Err() != nil {
    return ctx.Err()
}

err := c.RefreshState(ctx, tablet)
if err != nil && strings.Contains(err.Error(), "failed to acquire topoReadPool") {
    time.Sleep(backoff)
    err = c.RefreshState(ctx, tablet)
}

When it happens

Trigger: Calling Cluster.RefreshState (vtadmin RefreshState RPC) while topoReadPool is saturated with other topo reads, or after the request context has been cancelled/timed out.

Common situations: Bulk tablet refresh loops over many tablets exhausting the read pool; dashboards polling RefreshState concurrently; slow topo backend (etcd latency) making waiters time out.

Related errors

• PlannedFailoverShard(%s/%s): failed to acquire failoverPool:
• RefreshTabletReplicationSource(%v) failed to acquire topoRWP
• ReloadSchemas: failed to acquire topoReadPool: %w
• TabletExternallyPromoted(%s): failed to acquire topoRWPool:
• invalid choice for enum

AI-assisted analysis of vitessio/vitess@01a25a7d17 (2026-09-01). Data as JSON: /api/errors/498e098ce6c0fb22. Report an issue: GitHub.