vitessio/vitess · critical

EmergencyFailoverShard(%s/%s) failed to acquire emergencyFai

Error message

EmergencyFailoverShard(%s/%s) failed to acquire emergencyFailoverPool: %w

What it means

EmergencyFailoverShard acquires a dedicated emergencyFailoverPool slot (kept separate from regular topo pools so a failover is never starved by routine work). If Acquire(ctx) fails — pool busy and the context cancelled/timed out waiting — the error names the keyspace/shard and the pool. Only one emergency reparent should run per cluster at a time.

Source

Thrown at go/vt/vtadmin/cluster/cluster.go:581

// old primary is dead or otherwise not responding.
func (c *Cluster) EmergencyFailoverShard(ctx context.Context, req *vtctldatapb.EmergencyReparentShardRequest) (*vtadminpb.EmergencyFailoverShardResponse, error) {
	span, ctx := trace.NewSpan(ctx, "Cluster.EmergencyFailoverShard")
	defer span.Finish()

	AnnotateSpan(c, span)
	span.Annotate("keyspace", req.Keyspace)
	span.Annotate("shard", req.Shard)
	span.Annotate("new_primary", topoproto.TabletAliasString(req.NewPrimary))
	span.Annotate("ignore_replicas", strings.Join(topoproto.TabletAliasList(req.IgnoreReplicas).ToStringSlice(), ","))
	span.Annotate("prevent_cross_cell_promotion", req.PreventCrossCellPromotion)
	span.Annotate("wait_for_all_tablets", req.WaitForAllTablets)

	if d, ok, err := protoutil.DurationFromProto(req.WaitReplicasTimeout); ok && err == nil {
		span.Annotate("wait_replicas_timeout", d.String())
	}

	if err := c.emergencyFailoverPool.Acquire(ctx); err != nil {
		return nil, fmt.Errorf("EmergencyFailoverShard(%s/%s) failed to acquire emergencyFailoverPool: %w", req.Keyspace, req.Shard, err)
	}
	defer c.emergencyFailoverPool.Release()

	resp, err := c.Vtctld.EmergencyReparentShard(ctx, req)
	if err != nil {
		return nil, err
	}

	return &vtadminpb.EmergencyFailoverShardResponse{
		Cluster:         c.ToProto(),
		Keyspace:        resp.Keyspace,
		Shard:           resp.Shard,
		PromotedPrimary: resp.PromotedPrimary,
		Events:          resp.Events,
	}, nil
}

// FindAllShardsInKeyspaceOptions modify the behavior of a cluster's

View on GitHub (pinned to 01a25a7d17)

Solutions

1. Check whether an EmergencyReparentShard is already running; wait for it to finish instead of retrying immediately
2. Retry with a longer WaitReplicasTimeout / context deadline
3. If double-triggering automation is at fault, add dedup/lock in the calling tool
4. If legitimate concurrent failovers on different shards are needed, ensure the pool capacity covers them

Example fix

// before
ctx, cancel := context.WithTimeout(ctx, 2*time.Second)
defer cancel()
cluster.EmergencyFailoverShard(ctx, req) // times out waiting for pool
// after
ctx, cancel := context.WithTimeout(ctx, 60*time.Second)
defer cancel()
cluster.EmergencyFailoverShard(ctx, req)

if req == nil || req.Keyspace == "" || req.Shard == "" {
    return fmt.Errorf("EmergencyFailoverShard: keyspace and shard are required")
}
if alreadyRunning := failoverInFlight(req.Keyspace, req.Shard); alreadyRunning {
    return fmt.Errorf("emergency failover already in progress for %s/%s", req.Keyspace, req.Shard)
}

resp, err := cluster.EmergencyFailoverShard(ctx, req)
if err != nil {
    if strings.Contains(err.Error(), "failed to acquire emergencyFailoverPool") {
        log.Warn("emergency failover pool busy; a failover may already be running; not retrying automatically")
        return err // failover must not be blindly retried
    }
    return err
}

When it happens

Trigger: Issuing EmergencyFailoverShard while another emergency reparent is already in progress on the same cluster and the new context expires waiting; ctx cancelled during Acquire.

Common situations: Operators (or automation) triggering failover twice in quick succession during an incident; monitoring/alerting double-firing on the same shard outage; very short operational timeouts during incident response.

Related errors

• DeleteKeyspace(%+v) failed to acquire topoRWPool: %w
• DeleteShards(%+v) failed to acquire topoRWPool: %w
• DeleteTablets(%+v) failed to acquire topoRWPool: %w
• FindAllShardsInKeyspace(%s) failed to acquire topoReadPool:
• findWorkflows(keyspaces = %v, opts = %+v) failed to acquire

AI-assisted analysis of vitessio/vitess@01a25a7d17 (2026-09-01). Data as JSON: /api/errors/48241c8e8041e678. Report an issue: GitHub.