{"record":{"id":"d5c792e4097b9a3d","repo":"vitessio/vitess","slug":"plannedfailovershard-s-s-failed-to-acquire-fai","errorCode":null,"errorMessage":"PlannedFailoverShard(%s/%s): failed to acquire failoverPool: %w","messagePattern":"PlannedFailoverShard\\((.+?)/(.+?)\\): failed to acquire failoverPool: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"go/vt/vtadmin/cluster/cluster.go","lineNumber":2097,"sourceCode":"// PlannedFailoverShard fails over the shard either to a new primary or away\n// from an old primary. Both the current and candidate primaries must be\n// reachable and running.\nfunc (c *Cluster) PlannedFailoverShard(ctx context.Context, req *vtctldatapb.PlannedReparentShardRequest) (*vtadminpb.PlannedFailoverShardResponse, error) {\n\tspan, ctx := trace.NewSpan(ctx, \"Cluster.PlannedFailoverShard\")\n\tdefer span.Finish()\n\n\tAnnotateSpan(c, span)\n\tspan.Annotate(\"keyspace\", req.Keyspace)\n\tspan.Annotate(\"shard\", req.Shard)\n\tspan.Annotate(\"new_primary\", topoproto.TabletAliasString(req.NewPrimary))\n\tspan.Annotate(\"avoid_primary\", topoproto.TabletAliasString(req.AvoidPrimary))\n\n\tif d, ok, err := protoutil.DurationFromProto(req.WaitReplicasTimeout); ok && err == nil {\n\t\tspan.Annotate(\"wait_replicas_timeout\", d.String())\n\t}\n\n\tif err := c.failoverPool.Acquire(ctx); err != nil {\n\t\treturn nil, fmt.Errorf(\"PlannedFailoverShard(%s/%s): failed to acquire failoverPool: %w\", req.Keyspace, req.Shard, err)\n\t}\n\tdefer c.failoverPool.Release()\n\n\tresp, err := c.Vtctld.PlannedReparentShard(ctx, req)\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\n\treturn &vtadminpb.PlannedFailoverShardResponse{\n\t\tCluster:         c.ToProto(),\n\t\tKeyspace:        resp.Keyspace,\n\t\tShard:           resp.Shard,\n\t\tPromotedPrimary: resp.PromotedPrimary,\n\t\tEvents:          resp.Events,\n\t}, nil\n}\n\n// RefreshState reloads the tablet record from a cluster's topo on a tablet.","sourceCodeStart":2079,"sourceCodeEnd":2115,"githubUrl":"https://github.com/vitessio/vitess/blob/01a25a7d176f94613b8d59d799f438380a8760e4/go/vt/vtadmin/cluster/cluster.go#L2079-L2115","documentation":"Cluster.PlannedFailoverShard guards concurrent failovers with a semaphore (failoverPool). If ctx is cancelled/expired or the semaphore's buffer of concurrent failover slots is exhausted before Acquire succeeds, the acquisition error is wrapped and returned.","triggerScenarios":"Calling PlannedFailoverShard while the maximum number of concurrent PlannedReparentShard operations are already running; or with a ctx that is cancelled/timed out before Acquire completes.","commonSituations":"Multiple simultaneous shard failovers (e.g. automated failover scripts or dashboards firing in parallel); client-side timeouts during a planned failover; PRS taking longer than the caller's context deadline while others queue.","solutions":["Retry with a fresh, uncancelled context after in-flight failovers finish","Reduce concurrency of failover callers or serialize them client-side","Increase the caller's context timeout so Acquire does not time out while waiting","Increase cluster.failover_pool_size config if more parallel failovers are legitimately needed"],"exampleFix":"// before\nctx, cancel := context.WithTimeout(context.Background(), time.Second)\nresp, err := c.PlannedFailoverShard(ctx, req)\n// after\nctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)\nresp, err := c.PlannedFailoverShard(ctx, req)","handlingStrategy":"retry","validationCode":"if ctx.Err() != nil {\n    return fmt.Errorf(\"context already cancelled before failover: %w\", ctx.Err())\n}","typeGuard":null,"tryCatchPattern":"resp, err := c.PlannedFailoverShard(ctx, req)\nif err != nil && strings.Contains(err.Error(), \"failed to acquire failoverPool\") {\n    // back off and retry with a fresh context\n}","preventionTips":["Serialize failovers; never fire PlannedFailoverShard concurrently from automation","Use generous context timeouts (failovers can take minutes)","Size cluster.failover_pool_size to expected parallel failover demand"],"tags":["vtadmin","concurrency","semaphore","context-cancelled"],"backgroundTag":"semaphore-acquisition-failed","analyzedSha":"01a25a7d176f94613b8d59d799f438380a8760e4","analyzedAt":"2026-09-01T17:28:30.605Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}