vitessio/vitess · error

CreateShard(%+v) failed to acquire topoRWPool: %w

Error message

CreateShard(%+v) failed to acquire topoRWPool: %w

What it means

CreateShard must acquire the cluster's topo read-write pool (c.topoRWPool.Acquire(ctx)) before creating the shard. If acquisition fails — due to context cancellation/timeout while waiting, pool exhaustion, or topo backend unavailability — the error is wrapped with the full request for debugging.

Source

Thrown at go/vt/vtadmin/cluster/cluster.go:481

	if req == nil {
		return nil, fmt.Errorf("%w: request cannot be nil", errors.ErrInvalidRequest)
	}

	span.Annotate("keyspace", req.Keyspace)
	span.Annotate("shard", req.ShardName)
	span.Annotate("force", req.Force)
	span.Annotate("include_parent", req.IncludeParent)

	if req.Keyspace == "" {
		return nil, fmt.Errorf("%w: keyspace name is required", errors.ErrInvalidRequest)
	}

	if req.ShardName == "" {
		return nil, fmt.Errorf("%w: shard name is required", errors.ErrInvalidRequest)
	}

	if err := c.topoRWPool.Acquire(ctx); err != nil {
		return nil, fmt.Errorf("CreateShard(%+v) failed to acquire topoRWPool: %w", req, err)
	}
	defer c.topoRWPool.Release()

	return c.Vtctld.CreateShard(ctx, req)
}

// DeleteKeyspace deletes a keyspace in the given cluster, proxying a
// DeleteKeyspaceRequest to a vtctld in that cluster.
func (c *Cluster) DeleteKeyspace(ctx context.Context, req *vtctldatapb.DeleteKeyspaceRequest) (*vtctldatapb.DeleteKeyspaceResponse, error) {
	span, ctx := trace.NewSpan(ctx, "Cluster.DeleteKeyspace")
	defer span.Finish()

	AnnotateSpan(c, span)

	if req == nil {
		return nil, fmt.Errorf("%w: request cannot be nil", errors.ErrInvalidRequest)
	}

View on GitHub (pinned to 01a25a7d17)

Solutions

1. Increase the request timeout / use a context with an adequate deadline so Acquire can wait its turn.
2. Check etcd/zookeeper health and network connectivity from vtadmin.
3. Find operations stuck holding topoRWPool (stuck CreateShard/CreateKeyspace/ApplySchema calls) and clear them.
4. Retry once the topo backend is responsive; acquisition failures under contention are typically transient.

Example fix

// before
ctx, cancel := context.WithTimeout(ctx, 2*time.Second) // too short; Acquire times out
// after
ctx, cancel := context.WithTimeout(ctx, 30*time.Second)
defer cancel()
err := cluster.CreateShard(ctx, req)

ctx, cancel := context.WithTimeout(ctx, 30*time.Second)
defer cancel()
if err := cluster.TopoServer.Conn().Ping(ctx); err != nil {
	return fmt.Errorf("topo server unreachable: %w", err)
}

var resp *vtctldatapb.CreateShardResponse
err := retry.Do(func() error {
	var e error
	resp, e = cluster.CreateShard(ctx, req)
	return e
}, retry.Attempts(3), retry.RetryIf(func(err error) bool {
	return strings.Contains(err.Error(), "failed to acquire topoRWPool")
}))

When it happens

Trigger: Calling CreateShard when other operations hold topoRWPool, when ctx is cancelled/deadline-exceeded while waiting for the pool, or when etcd/zookeeper is unreachable.

Common situations: Concurrent bulk shard-creation scripts serializing on the pool and timing out; topo server outage or network partition; request deadlines shorter than pool wait time.

Related errors

• CreateKeyspace(%+v) failed to acquire topoRWPool: %w
• invalid choice for enum
• failed to parse tablet_alias %s: %w
• DeleteKeyspace(%+v) failed to acquire topoRWPool: %w
• DeleteShards(%+v) failed to acquire topoRWPool: %w

AI-assisted analysis of vitessio/vitess@01a25a7d17 (2026-09-01). Data as JSON: /api/errors/e7862cb6032701d1. Report an issue: GitHub.