{"record":{"id":"852a6ec24a68e28f","repo":"vitessio/vitess","slug":"tabletexternallypromoted-s-failed-to-acquire-to","errorCode":null,"errorMessage":"TabletExternallyPromoted(%s): failed to acquire topoRWPool: %w","messagePattern":"TabletExternallyPromoted\\((.+?)\\): failed to acquire topoRWPool: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"go/vt/vtadmin/cluster/cluster.go","lineNumber":2457,"sourceCode":"\tAnnotateSpan(c, span)\n\tspan.Annotate(\"tablet_alias\", topoproto.TabletAliasString(req.TabletAlias))\n\tspan.Annotate(\"writable\", req.Writable)\n\n\t_, err := c.Vtctld.SetWritable(ctx, req)\n\treturn err\n}\n\n// TabletExternallyPromoted updates the topo record for a shard to reflect a\n// tablet that was promoted to primary external to Vitess (e.g. orchestrator).\nfunc (c *Cluster) TabletExternallyPromoted(ctx context.Context, tablet *vtadminpb.Tablet) (*vtadminpb.TabletExternallyPromotedResponse, error) {\n\tspan, ctx := trace.NewSpan(ctx, \"API.TabletExternallyPromoted\")\n\tdefer span.Finish()\n\n\tAnnotateSpan(c, span)\n\tspan.Annotate(\"tablet_alias\", topoproto.TabletAliasString(tablet.Tablet.Alias))\n\n\tif err := c.topoRWPool.Acquire(ctx); err != nil {\n\t\treturn nil, fmt.Errorf(\"TabletExternallyPromoted(%s): failed to acquire topoRWPool: %w\", topoproto.TabletAliasString(tablet.Tablet.Alias), err)\n\t}\n\tdefer c.topoRWPool.Release()\n\n\tresp, err := c.Vtctld.TabletExternallyReparented(ctx, &vtctldatapb.TabletExternallyReparentedRequest{\n\t\tTablet: tablet.Tablet.Alias,\n\t})\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\n\treturn &vtadminpb.TabletExternallyPromotedResponse{\n\t\tCluster:    c.ToProto(),\n\t\tKeyspace:   resp.Keyspace,\n\t\tShard:      resp.Shard,\n\t\tNewPrimary: resp.NewPrimary,\n\t\tOldPrimary: resp.OldPrimary,\n\t}, nil\n}","sourceCodeStart":2439,"sourceCodeEnd":2475,"githubUrl":"https://github.com/vitessio/vitess/blob/01a25a7d176f94613b8d59d799f438380a8760e4/go/vt/vtadmin/cluster/cluster.go#L2439-L2475","documentation":"Cluster.TabletExternallyPromoted acquires topoRWPool before calling vtctld TabletExternallyReparented (an external-tools promotion like Orchestrator). If Acquire fails due to context cancellation or pool exhaustion, the error is wrapped with the tablet alias.","triggerScenarios":"Calling TabletExternallyPromoted while topoRWPool slots are held by other topo write operations (failovers, ReparentTablet), or with a cancelled/expired context.","commonSituations":"External promotion (e.g. via Orchestrator) issued while a planned/emergency failover is concurrently running; automation scripts promoting tablets in parallel; request deadline too short given pool contention.","solutions":["Retry the promotion with a fresh context once other topo writes finish","Ensure only one promotion path (vtadmin vs external tool) runs at a time","Increase topo_rw_pool_size in vtadmin config","Extend the caller's timeout to tolerate waiting for a pool slot"],"exampleFix":"// before\nctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)\nresp, err := c.TabletExternallyPromoted(ctx, tablet)\n// after\nctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)\nresp, err := c.TabletExternallyPromoted(ctx, tablet)","handlingStrategy":"retry","validationCode":"if ctx.Err() != nil {\n    return ctx.Err()\n}\n// ensure no failover is in flight before external promotion","typeGuard":null,"tryCatchPattern":"resp, err := c.TabletExternallyPromoted(ctx, tablet)\nif err != nil && strings.Contains(err.Error(), \"failed to acquire topoRWPool\") {\n    // wait for failover completion and retry with fresh context\n}","preventionTips":["Do not mix vtadmin promotions with external orchestrator failovers concurrently","Use long timeouts around external promotions","Monitor for stuck topo write operations holding pool slots"],"tags":["vtadmin","concurrency","failover","context-cancelled"],"backgroundTag":"semaphore-acquisition-failed","analyzedSha":"01a25a7d176f94613b8d59d799f438380a8760e4","analyzedAt":"2026-09-01T17:28:30.605Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}