{"record":{"id":"e7862cb6032701d1","repo":"vitessio/vitess","slug":"createshard-v-failed-to-acquire-toporwpool-w","errorCode":null,"errorMessage":"CreateShard(%+v) failed to acquire topoRWPool: %w","messagePattern":"CreateShard\\(%\\+v\\) failed to acquire topoRWPool: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"go/vt/vtadmin/cluster/cluster.go","lineNumber":481,"sourceCode":"\tif req == nil {\n\t\treturn nil, fmt.Errorf(\"%w: request cannot be nil\", errors.ErrInvalidRequest)\n\t}\n\n\tspan.Annotate(\"keyspace\", req.Keyspace)\n\tspan.Annotate(\"shard\", req.ShardName)\n\tspan.Annotate(\"force\", req.Force)\n\tspan.Annotate(\"include_parent\", req.IncludeParent)\n\n\tif req.Keyspace == \"\" {\n\t\treturn nil, fmt.Errorf(\"%w: keyspace name is required\", errors.ErrInvalidRequest)\n\t}\n\n\tif req.ShardName == \"\" {\n\t\treturn nil, fmt.Errorf(\"%w: shard name is required\", errors.ErrInvalidRequest)\n\t}\n\n\tif err := c.topoRWPool.Acquire(ctx); err != nil {\n\t\treturn nil, fmt.Errorf(\"CreateShard(%+v) failed to acquire topoRWPool: %w\", req, err)\n\t}\n\tdefer c.topoRWPool.Release()\n\n\treturn c.Vtctld.CreateShard(ctx, req)\n}\n\n// DeleteKeyspace deletes a keyspace in the given cluster, proxying a\n// DeleteKeyspaceRequest to a vtctld in that cluster.\nfunc (c *Cluster) DeleteKeyspace(ctx context.Context, req *vtctldatapb.DeleteKeyspaceRequest) (*vtctldatapb.DeleteKeyspaceResponse, error) {\n\tspan, ctx := trace.NewSpan(ctx, \"Cluster.DeleteKeyspace\")\n\tdefer span.Finish()\n\n\tAnnotateSpan(c, span)\n\n\tif req == nil {\n\t\treturn nil, fmt.Errorf(\"%w: request cannot be nil\", errors.ErrInvalidRequest)\n\t}\n","sourceCodeStart":463,"sourceCodeEnd":499,"githubUrl":"https://github.com/vitessio/vitess/blob/01a25a7d176f94613b8d59d799f438380a8760e4/go/vt/vtadmin/cluster/cluster.go#L463-L499","documentation":"CreateShard must acquire the cluster's topo read-write pool (c.topoRWPool.Acquire(ctx)) before creating the shard. If acquisition fails — due to context cancellation/timeout while waiting, pool exhaustion, or topo backend unavailability — the error is wrapped with the full request for debugging.","triggerScenarios":"Calling CreateShard when other operations hold topoRWPool, when ctx is cancelled/deadline-exceeded while waiting for the pool, or when etcd/zookeeper is unreachable.","commonSituations":"Concurrent bulk shard-creation scripts serializing on the pool and timing out; topo server outage or network partition; request deadlines shorter than pool wait time.","solutions":["Increase the request timeout / use a context with an adequate deadline so Acquire can wait its turn.","Check etcd/zookeeper health and network connectivity from vtadmin.","Find operations stuck holding topoRWPool (stuck CreateShard/CreateKeyspace/ApplySchema calls) and clear them.","Retry once the topo backend is responsive; acquisition failures under contention are typically transient."],"exampleFix":"// before\nctx, cancel := context.WithTimeout(ctx, 2*time.Second) // too short; Acquire times out\n// after\nctx, cancel := context.WithTimeout(ctx, 30*time.Second)\ndefer cancel()\nerr := cluster.CreateShard(ctx, req)","handlingStrategy":"retry","validationCode":"ctx, cancel := context.WithTimeout(ctx, 30*time.Second)\ndefer cancel()\nif err := cluster.TopoServer.Conn().Ping(ctx); err != nil {\n\treturn fmt.Errorf(\"topo server unreachable: %w\", err)\n}","typeGuard":null,"tryCatchPattern":"var resp *vtctldatapb.CreateShardResponse\nerr := retry.Do(func() error {\n\tvar e error\n\tresp, e = cluster.CreateShard(ctx, req)\n\treturn e\n}, retry.Attempts(3), retry.RetryIf(func(err error) bool {\n\treturn strings.Contains(err.Error(), \"failed to acquire topoRWPool\")\n}))","preventionTips":["Bound topo-mutating operations with adequate deadlines; contention on topoRWPool is common during bulk shard creation.","Throttle concurrent writers to the topo server.","Monitor etcd/zookeeper health; most Acquire failures trace back to the topo backend.","Always pass contexts with deadlines so hung Acquire calls fail fast rather than hang forever."],"tags":["vtadmin","topo","lock-acquisition","timeout"],"backgroundTag":"topo-lock-acquire-failed","analyzedSha":"01a25a7d176f94613b8d59d799f438380a8760e4","analyzedAt":"2026-09-01T17:28:30.605Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}