{"record":{"id":"488acb1686de03ad","repo":"vitessio/vitess","slug":"createkeyspace-v-failed-to-acquire-toporwpool","errorCode":null,"errorMessage":"CreateKeyspace(%+v) failed to acquire topoRWPool: %w","messagePattern":"CreateKeyspace\\(%\\+v\\) failed to acquire topoRWPool: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"go/vt/vtadmin/cluster/cluster.go","lineNumber":439,"sourceCode":"// CreateKeyspaceRequest to a vtctld in that cluster.\nfunc (c *Cluster) CreateKeyspace(ctx context.Context, req *vtctldatapb.CreateKeyspaceRequest) (*vtadminpb.Keyspace, error) {\n\tspan, ctx := trace.NewSpan(ctx, \"Cluster.CreateKeyspace\")\n\tdefer span.Finish()\n\n\tAnnotateSpan(c, span)\n\n\tif req == nil {\n\t\treturn nil, fmt.Errorf(\"%w: request cannot be nil\", errors.ErrInvalidRequest)\n\t}\n\n\tif req.Name == \"\" {\n\t\treturn nil, fmt.Errorf(\"%w: keyspace name is required\", errors.ErrInvalidRequest)\n\t}\n\n\tspan.Annotate(\"keyspace\", req.Name)\n\n\tif err := c.topoRWPool.Acquire(ctx); err != nil {\n\t\treturn nil, fmt.Errorf(\"CreateKeyspace(%+v) failed to acquire topoRWPool: %w\", req, err)\n\t}\n\tdefer c.topoRWPool.Release()\n\n\tresp, err := c.Vtctld.CreateKeyspace(ctx, req)\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\n\treturn &vtadminpb.Keyspace{\n\t\tCluster:  c.ToProto(),\n\t\tKeyspace: resp.Keyspace,\n\t\tShards:   map[string]*vtctldatapb.Shard{},\n\t}, nil\n}\n\n// CreateShard creates a shard in the given cluster, proxying a\n// CreateShardRequest to a vtctld in that cluster.\nfunc (c *Cluster) CreateShard(ctx context.Context, req *vtctldatapb.CreateShardRequest) (*vtctldatapb.CreateShardResponse, error) {","sourceCodeStart":421,"sourceCodeEnd":457,"githubUrl":"https://github.com/vitessio/vitess/blob/01a25a7d176f94613b8d59d799f438380a8760e4/go/vt/vtadmin/cluster/cluster.go#L421-L457","documentation":"CreateKeyspace must take a read-write lock on the topology server via c.topoRWPool.Acquire(ctx) before mutating it. If acquisition fails — most commonly because ctx is cancelled/timed out while waiting for the pool, or the topo connection is unhealthy — the error is wrapped with the keyspace request details.","triggerScenarios":"Calling CreateKeyspace when the topo RW pool is exhausted by other writers, when ctx is cancelled or times out while queued for the pool, or when the topo server (etcd2/zk2) is unreachable so pool initialization fails.","commonSituations":"Long-running concurrent keyspace/shard creation operations holding the pool; etcd cluster down or partitioned; request deadline (HTTP timeout, gRPC deadline) shorter than pool wait time.","solutions":["Check ctx cancellation/deadline; increase the request timeout so Acquire can wait for the pool.","Verify the topo backend (etcd/zookeeper) is healthy and reachable from vtadmin.","Look for other operations holding topoRWPool (stuck CreateKeyspace/CreateShard/ApplySchema) and resolve or restart them.","Retry the operation once the topo server is responsive — Acquire failures under load are often transient."],"exampleFix":"// before\nctx := context.Background() // no deadline; hangs then fails on topo issues\n// after\nctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)\ndefer cancel()\nresp, err := cluster.CreateKeyspace(ctx, req)","handlingStrategy":"retry","validationCode":"ctx, cancel := context.WithTimeout(ctx, 30*time.Second)\ndefer cancel()\n// pre-check topo connectivity before the call:\nif err := cluster.TopoServer.Conn().Ping(ctx); err != nil {\n\treturn fmt.Errorf(\"topo server unreachable: %w\", err)\n}","typeGuard":null,"tryCatchPattern":"var resp *vtctldatapb.CreateKeyspaceResponse\nerr := retry.Do(func() error {\n\tvar e error\n\tresp, e = cluster.CreateKeyspace(ctx, req)\n\treturn e\n}, retry.Attempts(3), retry.RetryIf(func(err error) bool {\n\treturn strings.Contains(err.Error(), \"failed to acquire topoRWPool\")\n}))","preventionTips":["Use generous timeouts for topo-mutating operations; pool contention makes them slow.","Keep concurrent topo writers (scripts, CI jobs) limited to avoid pool exhaustion.","Monitor topo backend (etcd/zk) health so outages are caught before they become Acquire failures.","Always pass a cancellable, deadline-bounded context."],"tags":["vtadmin","topo","lock-acquisition","timeout"],"backgroundTag":"topo-lock-acquire-failed","analyzedSha":"01a25a7d176f94613b8d59d799f438380a8760e4","analyzedAt":"2026-09-01T17:28:30.605Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}