vitessio/vitess · error

CreateKeyspace(%+v) failed to acquire topoRWPool: %w

Error message

CreateKeyspace(%+v) failed to acquire topoRWPool: %w

What it means

CreateKeyspace must take a read-write lock on the topology server via c.topoRWPool.Acquire(ctx) before mutating it. If acquisition fails — most commonly because ctx is cancelled/timed out while waiting for the pool, or the topo connection is unhealthy — the error is wrapped with the keyspace request details.

Source

Thrown at go/vt/vtadmin/cluster/cluster.go:439

// CreateKeyspaceRequest to a vtctld in that cluster.
func (c *Cluster) CreateKeyspace(ctx context.Context, req *vtctldatapb.CreateKeyspaceRequest) (*vtadminpb.Keyspace, error) {
	span, ctx := trace.NewSpan(ctx, "Cluster.CreateKeyspace")
	defer span.Finish()

	AnnotateSpan(c, span)

	if req == nil {
		return nil, fmt.Errorf("%w: request cannot be nil", errors.ErrInvalidRequest)
	}

	if req.Name == "" {
		return nil, fmt.Errorf("%w: keyspace name is required", errors.ErrInvalidRequest)
	}

	span.Annotate("keyspace", req.Name)

	if err := c.topoRWPool.Acquire(ctx); err != nil {
		return nil, fmt.Errorf("CreateKeyspace(%+v) failed to acquire topoRWPool: %w", req, err)
	}
	defer c.topoRWPool.Release()

	resp, err := c.Vtctld.CreateKeyspace(ctx, req)
	if err != nil {
		return nil, err
	}

	return &vtadminpb.Keyspace{
		Cluster:  c.ToProto(),
		Keyspace: resp.Keyspace,
		Shards:   map[string]*vtctldatapb.Shard{},
	}, nil
}

// CreateShard creates a shard in the given cluster, proxying a
// CreateShardRequest to a vtctld in that cluster.
func (c *Cluster) CreateShard(ctx context.Context, req *vtctldatapb.CreateShardRequest) (*vtctldatapb.CreateShardResponse, error) {

View on GitHub (pinned to 01a25a7d17)

Solutions

1. Check ctx cancellation/deadline; increase the request timeout so Acquire can wait for the pool.
2. Verify the topo backend (etcd/zookeeper) is healthy and reachable from vtadmin.
3. Look for other operations holding topoRWPool (stuck CreateKeyspace/CreateShard/ApplySchema) and resolve or restart them.
4. Retry the operation once the topo server is responsive — Acquire failures under load are often transient.

Example fix

// before
ctx := context.Background() // no deadline; hangs then fails on topo issues
// after
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
resp, err := cluster.CreateKeyspace(ctx, req)

ctx, cancel := context.WithTimeout(ctx, 30*time.Second)
defer cancel()
// pre-check topo connectivity before the call:
if err := cluster.TopoServer.Conn().Ping(ctx); err != nil {
	return fmt.Errorf("topo server unreachable: %w", err)
}

var resp *vtctldatapb.CreateKeyspaceResponse
err := retry.Do(func() error {
	var e error
	resp, e = cluster.CreateKeyspace(ctx, req)
	return e
}, retry.Attempts(3), retry.RetryIf(func(err error) bool {
	return strings.Contains(err.Error(), "failed to acquire topoRWPool")
}))

When it happens

Trigger: Calling CreateKeyspace when the topo RW pool is exhausted by other writers, when ctx is cancelled or times out while queued for the pool, or when the topo server (etcd2/zk2) is unreachable so pool initialization fails.

Common situations: Long-running concurrent keyspace/shard creation operations holding the pool; etcd cluster down or partitioned; request deadline (HTTP timeout, gRPC deadline) shorter than pool wait time.

Related errors

• CreateShard(%+v) failed to acquire topoRWPool: %w
• invalid choice for enum
• failed to parse tablet_alias %s: %w
• DeleteKeyspace(%+v) failed to acquire topoRWPool: %w
• DeleteShards(%+v) failed to acquire topoRWPool: %w

AI-assisted analysis of vitessio/vitess@01a25a7d17 (2026-09-01). Data as JSON: /api/errors/488acb1686de03ad. Report an issue: GitHub.