{"record":{"id":"078891212cf441b8","repo":"vitessio/vitess","slug":"deleteshards-v-failed-to-acquire-toporwpool-w","errorCode":null,"errorMessage":"DeleteShards(%+v) failed to acquire topoRWPool: %w","messagePattern":"DeleteShards\\(%\\+v\\) failed to acquire topoRWPool: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"go/vt/vtadmin/cluster/cluster.go","lineNumber":539,"sourceCode":"\n\tif req == nil {\n\t\treturn nil, fmt.Errorf(\"%w: request cannot be nil\", errors.ErrInvalidRequest)\n\t}\n\n\tshards := make([]string, len(req.Shards))\n\tfor i, shard := range req.Shards {\n\t\tshards[i] = fmt.Sprintf(\"%s/%s\", shard.Keyspace, shard.Name)\n\t}\n\n\tsort.Strings(shards)\n\n\tspan.Annotate(\"num_shards\", len(shards))\n\tspan.Annotate(\"shards\", strings.Join(shards, \", \"))\n\tspan.Annotate(\"recursive\", req.Recursive)\n\tspan.Annotate(\"even_if_serving\", req.EvenIfServing)\n\n\tif err := c.topoRWPool.Acquire(ctx); err != nil {\n\t\treturn nil, fmt.Errorf(\"DeleteShards(%+v) failed to acquire topoRWPool: %w\", req, err)\n\t}\n\tdefer c.topoRWPool.Release()\n\n\treturn c.Vtctld.DeleteShards(ctx, req)\n}\n\n// DeleteTablets deletes one or more tablets in the given cluster.\nfunc (c *Cluster) DeleteTablets(ctx context.Context, req *vtctldatapb.DeleteTabletsRequest) (*vtctldatapb.DeleteTabletsResponse, error) {\n\tspan, ctx := trace.NewSpan(ctx, \"Cluster.DeleteTablets\")\n\tdefer span.Finish()\n\n\tAnnotateSpan(c, span)\n\tspan.Annotate(\"tablet_aliases\", strings.Join(topoproto.TabletAliasList(req.TabletAliases).ToStringSlice(), \",\"))\n\n\tif err := c.topoRWPool.Acquire(ctx); err != nil {\n\t\treturn nil, fmt.Errorf(\"DeleteTablets(%+v) failed to acquire topoRWPool: %w\", req, err)\n\t}\n\tdefer c.topoRWPool.Release()","sourceCodeStart":521,"sourceCodeEnd":557,"githubUrl":"https://github.com/vitessio/vitess/blob/01a25a7d176f94613b8d59d799f438380a8760e4/go/vt/vtadmin/cluster/cluster.go#L521-L557","documentation":"DeleteShards acquires the read-write topology pool before forwarding to vtctld. Failure to Acquire(ctx) — pool exhausted with the context expiring/cancelling while waiting — is wrapped as this error with the full request for diagnosis. It bounds concurrent destructive shard deletions in the cluster.","triggerScenarios":"Concurrent DeleteShards/DeleteKeyspace/EmergencyFailoverShard calls saturating topoRWPool; ctx deadline or cancellation during the wait in Acquire.","commonSituations":"CI cleanup jobs deleting shards across many clusters in parallel; requests racing an in-progress PRS/E RS failover that holds the RW slot; too-short HTTP timeouts.","solutions":["Retry with a longer context deadline","Lower parallelism of shard deletion (batch sequentially per cluster)","Ensure no long-running topology mutations are holding the RW pool","Increase topoRWPool capacity if contention is persistent"],"exampleFix":"// before\nfor _, shard := range shards {\n    go deleteShard(shard) // saturates topoRWPool, Acquire times out\n}\n// after\nsem := make(chan struct{}, 2)\nfor _, shard := range shards {\n    sem <- struct{}{}\n    go func(s string) { defer func() { <-sem }(); deleteShard(s) }(shard)\n}","handlingStrategy":"retry","validationCode":"if err := ctx.Err(); err != nil {\n    return fmt.Errorf(\"context already cancelled: %w\", err)\n}","typeGuard":null,"tryCatchPattern":"err := cluster.DeleteShards(ctx, req)\nif err != nil && strings.Contains(err.Error(), \"failed to acquire topoRWPool\") {\n    // back off and retry with a longer deadline\n    time.Sleep(2 * time.Second)\n    retryCtx, cancel := context.WithTimeout(context.Background(), 60*time.Second)\n    defer cancel()\n    resp, err = cluster.DeleteShards(retryCtx, req)\n}","preventionTips":["Bound parallelism of shard deletions to a small worker pool","Avoid running bulk deletes during failovers or resharding","Use context.WithTimeout rather than bare Background contexts"],"tags":["resource-pool","timeout","concurrency","vtadmin"],"backgroundTag":"pool-acquisition-failed","analyzedSha":"01a25a7d176f94613b8d59d799f438380a8760e4","analyzedAt":"2026-09-01T17:28:30.605Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}