FindAllShardsInKeyspace(%s) failed to acquire topoReadPool:

Error message

FindAllShardsInKeyspace(%s) failed to acquire topoReadPool: %w

What it means

FindAllShardsInKeyspace acquires the read-only topology pool before querying vtctld, unless opts.skipPool is set (used for re-entrant calls already holding a slot). If Acquire(ctx) fails — read pool exhausted and the context cancelled/timed out waiting — this error is wrapped with the keyspace. The read pool has more capacity than the RW pool but is still bounded.

Source

Thrown at go/vt/vtadmin/cluster/cluster.go:623

	// outside this package.
	skipPool bool
}

// FindAllShardsInKeyspace proxies a FindAllShardsInKeyspace RPC to a cluster's
// vtctld, unpacking the response struct.
//
// It can also optionally ensure the vtctldclient has a valid connection before
// making the RPC call.
func (c *Cluster) FindAllShardsInKeyspace(ctx context.Context, keyspace string, opts FindAllShardsInKeyspaceOptions) (map[string]*vtctldatapb.Shard, error) {
	span, ctx := trace.NewSpan(ctx, "Cluster.FindAllShardsInKeyspace")
	defer span.Finish()

	AnnotateSpan(c, span)
	span.Annotate("keyspace", keyspace)

	if !opts.skipPool {
		if err := c.topoReadPool.Acquire(ctx); err != nil {
			return nil, fmt.Errorf("FindAllShardsInKeyspace(%s) failed to acquire topoReadPool: %w", keyspace, err)
		}
		defer c.topoReadPool.Release()
	}

	resp, err := c.Vtctld.FindAllShardsInKeyspace(ctx, &vtctldatapb.FindAllShardsInKeyspaceRequest{
		Keyspace: keyspace,
	})
	if err != nil {
		return nil, fmt.Errorf("FindAllShardsInKeyspace(cluster = %s, keyspace = %s) failed: %w", c.ID, keyspace, err)
	}

	return resp.Shards, nil
}

// FindTablet returns the first tablet in a given cluster that satisfies the filter function.
func (c *Cluster) FindTablet(ctx context.Context, filter func(*vtadminpb.Tablet) bool) (*vtadminpb.Tablet, error) {
	span, ctx := trace.NewSpan(ctx, "Cluster.FindTablet")
	defer span.Finish()

View on GitHub (pinned to 01a25a7d17)

Solutions

Retry the call with a longer context deadline
Reduce the number of concurrent keyspace-wide reads (cache results, batch keyspace lists)
If calling from within a function that already holds the read pool, use the opts.skipPool path
Increase topoReadPool capacity if the workload legitimately needs it

Example fix

// before
for _, ks := range allKeyspaces {
    go cluster.FindAllShardsInKeyspace(ctx, ks, nil) // 500 keyspaces, read pool exhausted
}
// after
g, gctx := errgroup.WithContext(ctx)
g.SetLimit(8)
for _, ks := range allKeyspaces {
    ks := ks
    g.Go(func() error { _, err := cluster.FindAllShardsInKeyspace(gctx, ks, nil); return err })
}

Defensive patterns

Strategy: retry

Validate before calling

if keyspace == "" {
    return fmt.Errorf("keyspace name is required")
}
if err := ctx.Err(); err != nil {
    return fmt.Errorf("context already cancelled: %w", err)
}

Try / catch

shards, err := cluster.FindAllShardsInKeyspace(ctx, ks, nil)
if err != nil && strings.Contains(err.Error(), "failed to acquire topoReadPool") {
    retryCtx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
    defer cancel()
    shards, err = cluster.FindAllShardsInKeyspace(retryCtx, ks, nil)
}

Prevention

Limit concurrency of keyspace-wide reads with an errgroup SetLimit
Pass opts with skipPool only when already holding the read pool
Cache shard listings for short periods on read-heavy dashboards

When it happens

Trigger: Heavy concurrent reads (GetWorkflows, GetKeyspace, schema queries) exhausting topoReadPool; ctx deadline exceeded or cancelled while blocked in Acquire; opts.skipPool false when the caller is not already holding a slot.

Common situations: Dashboards polling many keyspaces concurrently; vtadmin web UI burst on page load; long schema-refresh operations holding read slots.

Related errors

AI-assisted analysis of vitessio/vitess@01a25a7d17 (2026-09-01). Data as JSON: /api/errors/28bcee5c545e6226. Report an issue: GitHub.