What causes "DeleteShards(%+v) failed to acquire topoRWPool: %w" in vitessio/vitess?

Retry with a longer context deadline

DeleteShards(%+v) failed to acquire topoRWPool: %w

Error message

DeleteShards(%+v) failed to acquire topoRWPool: %w

What it means

DeleteShards acquires the read-write topology pool before forwarding to vtctld. Failure to Acquire(ctx) — pool exhausted with the context expiring/cancelling while waiting — is wrapped as this error with the full request for diagnosis. It bounds concurrent destructive shard deletions in the cluster.

Source

Thrown at go/vt/vtadmin/cluster/cluster.go:539

	if req == nil {
		return nil, fmt.Errorf("%w: request cannot be nil", errors.ErrInvalidRequest)
	}

	shards := make([]string, len(req.Shards))
	for i, shard := range req.Shards {
		shards[i] = fmt.Sprintf("%s/%s", shard.Keyspace, shard.Name)
	}

	sort.Strings(shards)

	span.Annotate("num_shards", len(shards))
	span.Annotate("shards", strings.Join(shards, ", "))
	span.Annotate("recursive", req.Recursive)
	span.Annotate("even_if_serving", req.EvenIfServing)

	if err := c.topoRWPool.Acquire(ctx); err != nil {
		return nil, fmt.Errorf("DeleteShards(%+v) failed to acquire topoRWPool: %w", req, err)
	}
	defer c.topoRWPool.Release()

	return c.Vtctld.DeleteShards(ctx, req)
}

// DeleteTablets deletes one or more tablets in the given cluster.
func (c *Cluster) DeleteTablets(ctx context.Context, req *vtctldatapb.DeleteTabletsRequest) (*vtctldatapb.DeleteTabletsResponse, error) {
	span, ctx := trace.NewSpan(ctx, "Cluster.DeleteTablets")
	defer span.Finish()

	AnnotateSpan(c, span)
	span.Annotate("tablet_aliases", strings.Join(topoproto.TabletAliasList(req.TabletAliases).ToStringSlice(), ","))

	if err := c.topoRWPool.Acquire(ctx); err != nil {
		return nil, fmt.Errorf("DeleteTablets(%+v) failed to acquire topoRWPool: %w", req, err)
	}
	defer c.topoRWPool.Release()

View on GitHub (pinned to 01a25a7d17)

Solutions

Retry with a longer context deadline
Lower parallelism of shard deletion (batch sequentially per cluster)
Ensure no long-running topology mutations are holding the RW pool
Increase topoRWPool capacity if contention is persistent

Example fix

// before
for _, shard := range shards {
    go deleteShard(shard) // saturates topoRWPool, Acquire times out
}
// after
sem := make(chan struct{}, 2)
for _, shard := range shards {
    sem <- struct{}{}
    go func(s string) { defer func() { <-sem }(); deleteShard(s) }(shard)
}

Defensive patterns

Strategy: retry

Validate before calling

if err := ctx.Err(); err != nil {
    return fmt.Errorf("context already cancelled: %w", err)
}

Try / catch

err := cluster.DeleteShards(ctx, req)
if err != nil && strings.Contains(err.Error(), "failed to acquire topoRWPool") {
    // back off and retry with a longer deadline
    time.Sleep(2 * time.Second)
    retryCtx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
    defer cancel()
    resp, err = cluster.DeleteShards(retryCtx, req)
}

Prevention

Bound parallelism of shard deletions to a small worker pool
Avoid running bulk deletes during failovers or resharding
Use context.WithTimeout rather than bare Background contexts

When it happens

Trigger: Concurrent DeleteShards/DeleteKeyspace/EmergencyFailoverShard calls saturating topoRWPool; ctx deadline or cancellation during the wait in Acquire.

Common situations: CI cleanup jobs deleting shards across many clusters in parallel; requests racing an in-progress PRS/E RS failover that holds the RW slot; too-short HTTP timeouts.

Related errors

AI-assisted analysis of vitessio/vitess@01a25a7d17 (2026-09-01). Data as JSON: /api/errors/078891212cf441b8. Report an issue: GitHub.