vitessio/vitess · error

PlannedFailoverShard(%s/%s): failed to acquire failoverPool:

Error message

PlannedFailoverShard(%s/%s): failed to acquire failoverPool: %w

What it means

Cluster.PlannedFailoverShard guards concurrent failovers with a semaphore (failoverPool). If ctx is cancelled/expired or the semaphore's buffer of concurrent failover slots is exhausted before Acquire succeeds, the acquisition error is wrapped and returned.

Source

Thrown at go/vt/vtadmin/cluster/cluster.go:2097

// PlannedFailoverShard fails over the shard either to a new primary or away
// from an old primary. Both the current and candidate primaries must be
// reachable and running.
func (c *Cluster) PlannedFailoverShard(ctx context.Context, req *vtctldatapb.PlannedReparentShardRequest) (*vtadminpb.PlannedFailoverShardResponse, error) {
	span, ctx := trace.NewSpan(ctx, "Cluster.PlannedFailoverShard")
	defer span.Finish()

	AnnotateSpan(c, span)
	span.Annotate("keyspace", req.Keyspace)
	span.Annotate("shard", req.Shard)
	span.Annotate("new_primary", topoproto.TabletAliasString(req.NewPrimary))
	span.Annotate("avoid_primary", topoproto.TabletAliasString(req.AvoidPrimary))

	if d, ok, err := protoutil.DurationFromProto(req.WaitReplicasTimeout); ok && err == nil {
		span.Annotate("wait_replicas_timeout", d.String())
	}

	if err := c.failoverPool.Acquire(ctx); err != nil {
		return nil, fmt.Errorf("PlannedFailoverShard(%s/%s): failed to acquire failoverPool: %w", req.Keyspace, req.Shard, err)
	}
	defer c.failoverPool.Release()

	resp, err := c.Vtctld.PlannedReparentShard(ctx, req)
	if err != nil {
		return nil, err
	}

	return &vtadminpb.PlannedFailoverShardResponse{
		Cluster:         c.ToProto(),
		Keyspace:        resp.Keyspace,
		Shard:           resp.Shard,
		PromotedPrimary: resp.PromotedPrimary,
		Events:          resp.Events,
	}, nil
}

// RefreshState reloads the tablet record from a cluster's topo on a tablet.

View on GitHub (pinned to 01a25a7d17)

Solutions

1. Retry with a fresh, uncancelled context after in-flight failovers finish
2. Reduce concurrency of failover callers or serialize them client-side
3. Increase the caller's context timeout so Acquire does not time out while waiting
4. Increase cluster.failover_pool_size config if more parallel failovers are legitimately needed

Example fix

// before
ctx, cancel := context.WithTimeout(context.Background(), time.Second)
resp, err := c.PlannedFailoverShard(ctx, req)
// after
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
resp, err := c.PlannedFailoverShard(ctx, req)

if ctx.Err() != nil {
    return fmt.Errorf("context already cancelled before failover: %w", ctx.Err())
}

resp, err := c.PlannedFailoverShard(ctx, req)
if err != nil && strings.Contains(err.Error(), "failed to acquire failoverPool") {
    // back off and retry with a fresh context
}

When it happens

Trigger: Calling PlannedFailoverShard while the maximum number of concurrent PlannedReparentShard operations are already running; or with a ctx that is cancelled/timed out before Acquire completes.

Common situations: Multiple simultaneous shard failovers (e.g. automated failover scripts or dashboards firing in parallel); client-side timeouts during a planned failover; PRS taking longer than the caller's context deadline while others queue.

Related errors

• GetWorkflows(keyspace = %s, active_only = %v) failed to acqu
• GetBackups(%s/%s) failed to acquire backupReadPool: %w
• GetCellInfo(%s) failed to acquire topoReadPool: %w
• GetKeyspace(%s) failed to acquire topoReadPool: %w
• GetKeyspaces() failed to acquire topoReadPool: %w

AI-assisted analysis of vitessio/vitess@01a25a7d17 (2026-09-01). Data as JSON: /api/errors/d5c792e4097b9a3d. Report an issue: GitHub.