What causes "DeleteKeyspace(%+v) failed to acquire topoRWPool: %w" in vitessio/vitess?

Retry with a longer context deadline so Acquire can obtain a slot

DeleteKeyspace(%+v) failed to acquire topoRWPool: %w

Error message

DeleteKeyspace(%+v) failed to acquire topoRWPool: %w

What it means

DeleteKeyspace must acquire a read-write topology pool slot before mutating topo state. If c.topoRWPool.Acquire(ctx) fails — typically because the context was cancelled or timed out while all RW slots were in use — the error is wrapped with the request details. This serializes destructive topology operations against a bounded semaphore.

Source

Thrown at go/vt/vtadmin/cluster/cluster.go:507

// DeleteKeyspaceRequest to a vtctld in that cluster.
func (c *Cluster) DeleteKeyspace(ctx context.Context, req *vtctldatapb.DeleteKeyspaceRequest) (*vtctldatapb.DeleteKeyspaceResponse, error) {
	span, ctx := trace.NewSpan(ctx, "Cluster.DeleteKeyspace")
	defer span.Finish()

	AnnotateSpan(c, span)

	if req == nil {
		return nil, fmt.Errorf("%w: request cannot be nil", errors.ErrInvalidRequest)
	}

	if req.Keyspace == "" {
		return nil, fmt.Errorf("%w: keyspace name is required", errors.ErrInvalidRequest)
	}

	span.Annotate("keyspace", req.Keyspace)

	if err := c.topoRWPool.Acquire(ctx); err != nil {
		return nil, fmt.Errorf("DeleteKeyspace(%+v) failed to acquire topoRWPool: %w", req, err)
	}
	defer c.topoRWPool.Release()

	return c.Vtctld.DeleteKeyspace(ctx, req)
}

// DeleteShards deletes one or more shards in the given cluster, proxying a
// single DeleteShardsRequest to a vtctld in that cluster.
func (c *Cluster) DeleteShards(ctx context.Context, req *vtctldatapb.DeleteShardsRequest) (*vtctldatapb.DeleteShardsResponse, error) {
	span, ctx := trace.NewSpan(ctx, "Cluster.DeleteShards")
	defer span.Finish()

	AnnotateSpan(c, span)

	if req == nil {
		return nil, fmt.Errorf("%w: request cannot be nil", errors.ErrInvalidRequest)
	}

View on GitHub (pinned to 01a25a7d17)

Solutions

Retry with a longer context deadline so Acquire can obtain a slot
Reduce concurrency of topology-mutating calls (serialize bulk operations)
Check for in-flight long-running topology operations (failovers, migrations) that hold RW slots
If chronic, increase the cluster's topoRWPool capacity configuration

Example fix

// before
ctx := context.Background()
cluster.DeleteKeyspace(ctx, req) // cancelled while waiting for pool
// after
ctx, cancel := context.WithTimeout(ctx, 30*time.Second)
defer cancel()
cluster.DeleteKeyspace(ctx, req)

Defensive patterns

Strategy: retry

Validate before calling

if err := ctx.Err(); err != nil {
    return fmt.Errorf("context already cancelled: %w", err)
}

Try / catch

err := cluster.DeleteKeyspace(ctx, req)
if err != nil && strings.Contains(err.Error(), "failed to acquire topoRWPool") {
    retryCtx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
    defer cancel()
    resp, err = cluster.DeleteKeyspace(retryCtx, req)
}

Prevention

Use generous context timeouts for destructive topology operations
Serialize or rate-limit concurrent topology mutations in automation
Monitor pool wait times; alert on chronic contention

When it happens

Trigger: Many concurrent topology-mutating vtadmin calls (DeleteKeyspace/DeleteShards/DeleteTablets/EmergencyFailoverShard) exhausting topoRWPool capacity while others wait; ctx cancelled (client disconnect, deadline exceeded) while blocked in Acquire.

Common situations: Bulk deletion scripts hitting multiple clusters concurrently; long-running reshard or failover operations holding RW slots; HTTP request timeouts shorter than the wait for a pool slot.

Related errors

AI-assisted analysis of vitessio/vitess@01a25a7d17 (2026-09-01). Data as JSON: /api/errors/54f560c57ca9f320. Report an issue: GitHub.