{"record":{"id":"54f560c57ca9f320","repo":"vitessio/vitess","slug":"deletekeyspace-v-failed-to-acquire-toporwpool","errorCode":null,"errorMessage":"DeleteKeyspace(%+v) failed to acquire topoRWPool: %w","messagePattern":"DeleteKeyspace\\(%\\+v\\) failed to acquire topoRWPool: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"go/vt/vtadmin/cluster/cluster.go","lineNumber":507,"sourceCode":"// DeleteKeyspaceRequest to a vtctld in that cluster.\nfunc (c *Cluster) DeleteKeyspace(ctx context.Context, req *vtctldatapb.DeleteKeyspaceRequest) (*vtctldatapb.DeleteKeyspaceResponse, error) {\n\tspan, ctx := trace.NewSpan(ctx, \"Cluster.DeleteKeyspace\")\n\tdefer span.Finish()\n\n\tAnnotateSpan(c, span)\n\n\tif req == nil {\n\t\treturn nil, fmt.Errorf(\"%w: request cannot be nil\", errors.ErrInvalidRequest)\n\t}\n\n\tif req.Keyspace == \"\" {\n\t\treturn nil, fmt.Errorf(\"%w: keyspace name is required\", errors.ErrInvalidRequest)\n\t}\n\n\tspan.Annotate(\"keyspace\", req.Keyspace)\n\n\tif err := c.topoRWPool.Acquire(ctx); err != nil {\n\t\treturn nil, fmt.Errorf(\"DeleteKeyspace(%+v) failed to acquire topoRWPool: %w\", req, err)\n\t}\n\tdefer c.topoRWPool.Release()\n\n\treturn c.Vtctld.DeleteKeyspace(ctx, req)\n}\n\n// DeleteShards deletes one or more shards in the given cluster, proxying a\n// single DeleteShardsRequest to a vtctld in that cluster.\nfunc (c *Cluster) DeleteShards(ctx context.Context, req *vtctldatapb.DeleteShardsRequest) (*vtctldatapb.DeleteShardsResponse, error) {\n\tspan, ctx := trace.NewSpan(ctx, \"Cluster.DeleteShards\")\n\tdefer span.Finish()\n\n\tAnnotateSpan(c, span)\n\n\tif req == nil {\n\t\treturn nil, fmt.Errorf(\"%w: request cannot be nil\", errors.ErrInvalidRequest)\n\t}\n","sourceCodeStart":489,"sourceCodeEnd":525,"githubUrl":"https://github.com/vitessio/vitess/blob/01a25a7d176f94613b8d59d799f438380a8760e4/go/vt/vtadmin/cluster/cluster.go#L489-L525","documentation":"DeleteKeyspace must acquire a read-write topology pool slot before mutating topo state. If c.topoRWPool.Acquire(ctx) fails — typically because the context was cancelled or timed out while all RW slots were in use — the error is wrapped with the request details. This serializes destructive topology operations against a bounded semaphore.","triggerScenarios":"Many concurrent topology-mutating vtadmin calls (DeleteKeyspace/DeleteShards/DeleteTablets/EmergencyFailoverShard) exhausting topoRWPool capacity while others wait; ctx cancelled (client disconnect, deadline exceeded) while blocked in Acquire.","commonSituations":"Bulk deletion scripts hitting multiple clusters concurrently; long-running reshard or failover operations holding RW slots; HTTP request timeouts shorter than the wait for a pool slot.","solutions":["Retry with a longer context deadline so Acquire can obtain a slot","Reduce concurrency of topology-mutating calls (serialize bulk operations)","Check for in-flight long-running topology operations (failovers, migrations) that hold RW slots","If chronic, increase the cluster's topoRWPool capacity configuration"],"exampleFix":"// before\nctx := context.Background()\ncluster.DeleteKeyspace(ctx, req) // cancelled while waiting for pool\n// after\nctx, cancel := context.WithTimeout(ctx, 30*time.Second)\ndefer cancel()\ncluster.DeleteKeyspace(ctx, req)","handlingStrategy":"retry","validationCode":"if err := ctx.Err(); err != nil {\n    return fmt.Errorf(\"context already cancelled: %w\", err)\n}","typeGuard":null,"tryCatchPattern":"err := cluster.DeleteKeyspace(ctx, req)\nif err != nil && strings.Contains(err.Error(), \"failed to acquire topoRWPool\") {\n    retryCtx, cancel := context.WithTimeout(context.Background(), 60*time.Second)\n    defer cancel()\n    resp, err = cluster.DeleteKeyspace(retryCtx, req)\n}","preventionTips":["Use generous context timeouts for destructive topology operations","Serialize or rate-limit concurrent topology mutations in automation","Monitor pool wait times; alert on chronic contention"],"tags":["resource-pool","timeout","concurrency","vtadmin"],"backgroundTag":"pool-acquisition-failed","analyzedSha":"01a25a7d176f94613b8d59d799f438380a8760e4","analyzedAt":"2026-09-01T17:28:30.605Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}