vitessio/vitess · error

TabletExternallyPromoted(%s): failed to acquire topoRWPool:

Error message

TabletExternallyPromoted(%s): failed to acquire topoRWPool: %w

What it means

Cluster.TabletExternallyPromoted acquires topoRWPool before calling vtctld TabletExternallyReparented (an external-tools promotion like Orchestrator). If Acquire fails due to context cancellation or pool exhaustion, the error is wrapped with the tablet alias.

Source

Thrown at go/vt/vtadmin/cluster/cluster.go:2457

	AnnotateSpan(c, span)
	span.Annotate("tablet_alias", topoproto.TabletAliasString(req.TabletAlias))
	span.Annotate("writable", req.Writable)

	_, err := c.Vtctld.SetWritable(ctx, req)
	return err
}

// TabletExternallyPromoted updates the topo record for a shard to reflect a
// tablet that was promoted to primary external to Vitess (e.g. orchestrator).
func (c *Cluster) TabletExternallyPromoted(ctx context.Context, tablet *vtadminpb.Tablet) (*vtadminpb.TabletExternallyPromotedResponse, error) {
	span, ctx := trace.NewSpan(ctx, "API.TabletExternallyPromoted")
	defer span.Finish()

	AnnotateSpan(c, span)
	span.Annotate("tablet_alias", topoproto.TabletAliasString(tablet.Tablet.Alias))

	if err := c.topoRWPool.Acquire(ctx); err != nil {
		return nil, fmt.Errorf("TabletExternallyPromoted(%s): failed to acquire topoRWPool: %w", topoproto.TabletAliasString(tablet.Tablet.Alias), err)
	}
	defer c.topoRWPool.Release()

	resp, err := c.Vtctld.TabletExternallyReparented(ctx, &vtctldatapb.TabletExternallyReparentedRequest{
		Tablet: tablet.Tablet.Alias,
	})
	if err != nil {
		return nil, err
	}

	return &vtadminpb.TabletExternallyPromotedResponse{
		Cluster:    c.ToProto(),
		Keyspace:   resp.Keyspace,
		Shard:      resp.Shard,
		NewPrimary: resp.NewPrimary,
		OldPrimary: resp.OldPrimary,
	}, nil
}

View on GitHub (pinned to 01a25a7d17)

Solutions

1. Retry the promotion with a fresh context once other topo writes finish
2. Ensure only one promotion path (vtadmin vs external tool) runs at a time
3. Increase topo_rw_pool_size in vtadmin config
4. Extend the caller's timeout to tolerate waiting for a pool slot

Example fix

// before
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
resp, err := c.TabletExternallyPromoted(ctx, tablet)
// after
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
resp, err := c.TabletExternallyPromoted(ctx, tablet)

if ctx.Err() != nil {
    return ctx.Err()
}
// ensure no failover is in flight before external promotion

resp, err := c.TabletExternallyPromoted(ctx, tablet)
if err != nil && strings.Contains(err.Error(), "failed to acquire topoRWPool") {
    // wait for failover completion and retry with fresh context
}

When it happens

Trigger: Calling TabletExternallyPromoted while topoRWPool slots are held by other topo write operations (failovers, ReparentTablet), or with a cancelled/expired context.

Common situations: External promotion (e.g. via Orchestrator) issued while a planned/emergency failover is concurrently running; automation scripts promoting tablets in parallel; request deadline too short given pool contention.

Related errors

• PlannedFailoverShard(%s/%s): failed to acquire failoverPool:
• RefreshState(%v) failed to acquire topoReadPool: %w
• RefreshTabletReplicationSource(%v) failed to acquire topoRWP
• invalid choice for enum
• value must be either a float64 (interpreted as seconds) or a

AI-assisted analysis of vitessio/vitess@01a25a7d17 (2026-09-01). Data as JSON: /api/errors/852a6ec24a68e28f. Report an issue: GitHub.