vitessio/vitess · error

DeleteTablets(%+v) failed to acquire topoRWPool: %w

Error message

DeleteTablets(%+v) failed to acquire topoRWPool: %w

What it means

DeleteTablets acquires the read-write topology pool before deleting tablet records via vtctld. If c.topoRWPool.Acquire(ctx) fails, the error is wrapped with the request (including the tablet aliases being deleted). The pool bounds concurrent topology writes across the vtadmin cluster handle.

Source

Thrown at go/vt/vtadmin/cluster/cluster.go:555

	if err := c.topoRWPool.Acquire(ctx); err != nil {
		return nil, fmt.Errorf("DeleteShards(%+v) failed to acquire topoRWPool: %w", req, err)
	}
	defer c.topoRWPool.Release()

	return c.Vtctld.DeleteShards(ctx, req)
}

// DeleteTablets deletes one or more tablets in the given cluster.
func (c *Cluster) DeleteTablets(ctx context.Context, req *vtctldatapb.DeleteTabletsRequest) (*vtctldatapb.DeleteTabletsResponse, error) {
	span, ctx := trace.NewSpan(ctx, "Cluster.DeleteTablets")
	defer span.Finish()

	AnnotateSpan(c, span)
	span.Annotate("tablet_aliases", strings.Join(topoproto.TabletAliasList(req.TabletAliases).ToStringSlice(), ","))

	if err := c.topoRWPool.Acquire(ctx); err != nil {
		return nil, fmt.Errorf("DeleteTablets(%+v) failed to acquire topoRWPool: %w", req, err)
	}
	defer c.topoRWPool.Release()

	return c.Vtctld.DeleteTablets(ctx, req)
}

// EmergencyFailoverShard fails over a shard to a new primary. It assumes the
// old primary is dead or otherwise not responding.
func (c *Cluster) EmergencyFailoverShard(ctx context.Context, req *vtctldatapb.EmergencyReparentShardRequest) (*vtadminpb.EmergencyFailoverShardResponse, error) {
	span, ctx := trace.NewSpan(ctx, "Cluster.EmergencyFailoverShard")
	defer span.Finish()

	AnnotateSpan(c, span)
	span.Annotate("keyspace", req.Keyspace)
	span.Annotate("shard", req.Shard)
	span.Annotate("new_primary", topoproto.TabletAliasString(req.NewPrimary))
	span.Annotate("ignore_replicas", strings.Join(topoproto.TabletAliasList(req.IgnoreReplicas).ToStringSlice(), ","))
	span.Annotate("prevent_cross_cell_promotion", req.PreventCrossCellPromotion)

View on GitHub (pinned to 01a25a7d17)

Solutions

1. Retry with an extended context deadline
2. Throttle/serialize tablet deletions instead of fanning out
3. Check for concurrent topology-mutating operations in the cluster
4. Increase topoRWPool size if this recurs under normal load

Example fix

// before
for _, alias := range aliases {
    go deleteTablet(alias) // Acquire fails under contention
}
// after
for _, alias := range aliases {
    if err := deleteTablet(alias); err != nil {
        return err // sequential: no pool contention
    }
}

if req == nil || len(req.TabletAliases) == 0 {
    return fmt.Errorf("DeleteTablets: request and tablet aliases are required")
}

func isValidDeleteTabletsRequest(req *vtctldatapb.DeleteTabletsRequest) bool {
    return req != nil && len(req.TabletAliases) > 0
}

err := cluster.DeleteTablets(ctx, req)
if err != nil && strings.Contains(err.Error(), "failed to acquire topoRWPool") {
    retryCtx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
    defer cancel()
    resp, err = cluster.DeleteTablets(retryCtx, req)
}

When it happens

Trigger: Calling DeleteTablets while the topoRWPool is fully utilized by other destructive operations, with the context expiring before a slot frees; ctx cancellation (client disconnect) during the wait.

Common situations: Large-scale decommissioning scripts removing hundreds of tablets concurrently; overlap with an EmergencyFailoverShard or shard deletion; deadline-driven batch jobs.

Related errors

• DeleteKeyspace(%+v) failed to acquire topoRWPool: %w
• DeleteShards(%+v) failed to acquire topoRWPool: %w
• FindAllShardsInKeyspace(%s) failed to acquire topoReadPool:
• findWorkflows(keyspaces = %v, opts = %+v) failed to acquire
• invalid choice for enum

AI-assisted analysis of vitessio/vitess@01a25a7d17 (2026-09-01). Data as JSON: /api/errors/9ab21433ac33dc64. Report an issue: GitHub.