{"record":{"id":"48241c8e8041e678","repo":"vitessio/vitess","slug":"emergencyfailovershard-s-s-failed-to-acquire-em","errorCode":null,"errorMessage":"EmergencyFailoverShard(%s/%s) failed to acquire emergencyFailoverPool: %w","messagePattern":"EmergencyFailoverShard\\((.+?)/(.+?)\\) failed to acquire emergencyFailoverPool: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"go/vt/vtadmin/cluster/cluster.go","lineNumber":581,"sourceCode":"// old primary is dead or otherwise not responding.\nfunc (c *Cluster) EmergencyFailoverShard(ctx context.Context, req *vtctldatapb.EmergencyReparentShardRequest) (*vtadminpb.EmergencyFailoverShardResponse, error) {\n\tspan, ctx := trace.NewSpan(ctx, \"Cluster.EmergencyFailoverShard\")\n\tdefer span.Finish()\n\n\tAnnotateSpan(c, span)\n\tspan.Annotate(\"keyspace\", req.Keyspace)\n\tspan.Annotate(\"shard\", req.Shard)\n\tspan.Annotate(\"new_primary\", topoproto.TabletAliasString(req.NewPrimary))\n\tspan.Annotate(\"ignore_replicas\", strings.Join(topoproto.TabletAliasList(req.IgnoreReplicas).ToStringSlice(), \",\"))\n\tspan.Annotate(\"prevent_cross_cell_promotion\", req.PreventCrossCellPromotion)\n\tspan.Annotate(\"wait_for_all_tablets\", req.WaitForAllTablets)\n\n\tif d, ok, err := protoutil.DurationFromProto(req.WaitReplicasTimeout); ok && err == nil {\n\t\tspan.Annotate(\"wait_replicas_timeout\", d.String())\n\t}\n\n\tif err := c.emergencyFailoverPool.Acquire(ctx); err != nil {\n\t\treturn nil, fmt.Errorf(\"EmergencyFailoverShard(%s/%s) failed to acquire emergencyFailoverPool: %w\", req.Keyspace, req.Shard, err)\n\t}\n\tdefer c.emergencyFailoverPool.Release()\n\n\tresp, err := c.Vtctld.EmergencyReparentShard(ctx, req)\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\n\treturn &vtadminpb.EmergencyFailoverShardResponse{\n\t\tCluster:         c.ToProto(),\n\t\tKeyspace:        resp.Keyspace,\n\t\tShard:           resp.Shard,\n\t\tPromotedPrimary: resp.PromotedPrimary,\n\t\tEvents:          resp.Events,\n\t}, nil\n}\n\n// FindAllShardsInKeyspaceOptions modify the behavior of a cluster's","sourceCodeStart":563,"sourceCodeEnd":599,"githubUrl":"https://github.com/vitessio/vitess/blob/01a25a7d176f94613b8d59d799f438380a8760e4/go/vt/vtadmin/cluster/cluster.go#L563-L599","documentation":"EmergencyFailoverShard acquires a dedicated emergencyFailoverPool slot (kept separate from regular topo pools so a failover is never starved by routine work). If Acquire(ctx) fails — pool busy and the context cancelled/timed out waiting — the error names the keyspace/shard and the pool. Only one emergency reparent should run per cluster at a time.","triggerScenarios":"Issuing EmergencyFailoverShard while another emergency reparent is already in progress on the same cluster and the new context expires waiting; ctx cancelled during Acquire.","commonSituations":"Operators (or automation) triggering failover twice in quick succession during an incident; monitoring/alerting double-firing on the same shard outage; very short operational timeouts during incident response.","solutions":["Check whether an EmergencyReparentShard is already running; wait for it to finish instead of retrying immediately","Retry with a longer WaitReplicasTimeout / context deadline","If double-triggering automation is at fault, add dedup/lock in the calling tool","If legitimate concurrent failovers on different shards are needed, ensure the pool capacity covers them"],"exampleFix":"// before\nctx, cancel := context.WithTimeout(ctx, 2*time.Second)\ndefer cancel()\ncluster.EmergencyFailoverShard(ctx, req) // times out waiting for pool\n// after\nctx, cancel := context.WithTimeout(ctx, 60*time.Second)\ndefer cancel()\ncluster.EmergencyFailoverShard(ctx, req)","handlingStrategy":"try-catch","validationCode":"if req == nil || req.Keyspace == \"\" || req.Shard == \"\" {\n    return fmt.Errorf(\"EmergencyFailoverShard: keyspace and shard are required\")\n}\nif alreadyRunning := failoverInFlight(req.Keyspace, req.Shard); alreadyRunning {\n    return fmt.Errorf(\"emergency failover already in progress for %s/%s\", req.Keyspace, req.Shard)\n}","typeGuard":null,"tryCatchPattern":"resp, err := cluster.EmergencyFailoverShard(ctx, req)\nif err != nil {\n    if strings.Contains(err.Error(), \"failed to acquire emergencyFailoverPool\") {\n        log.Warn(\"emergency failover pool busy; a failover may already be running; not retrying automatically\")\n        return err // failover must not be blindly retried\n    }\n    return err\n}","preventionTips":["Add a distributed lock/dedup around failover triggers in automation","Never auto-retry emergency failovers without checking cluster state first","Use long deadlines during incidents; failovers can take minutes","Alert on pool acquisition failures — they signal concurrent failover attempts"],"tags":["resource-pool","failover","timeout","incident-response","vtadmin"],"backgroundTag":"pool-acquisition-failed","analyzedSha":"01a25a7d176f94613b8d59d799f438380a8760e4","analyzedAt":"2026-09-01T17:28:30.605Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}