vitessio/vitess · error

RefreshTabletReplicationSource(%v) failed to acquire topoRWP

Error message

RefreshTabletReplicationSource(%v) failed to acquire topoRWPool: %w

What it means

Cluster.RefreshTabletReplicationSource (ReparentTablet) acquires topoRWPool before issuing a write to the topo. Acquire failure — cancelled context or pool exhaustion — is wrapped with the tablet alias and returned to the caller.

Source

Thrown at go/vt/vtadmin/cluster/cluster.go:2144

	defer c.topoReadPool.Release()

	_, err := c.Vtctld.RefreshState(ctx, &vtctldatapb.RefreshStateRequest{
		TabletAlias: tablet.Tablet.Alias,
	})
	return err
}

// RefreshTabletReplicationSource performs a `CHANGE REPLICATION SOURCE TO` on
// a tablet to replicate from the current primary in the shard.
func (c *Cluster) RefreshTabletReplicationSource(ctx context.Context, tablet *vtadminpb.Tablet) (*vtadminpb.RefreshTabletReplicationSourceResponse, error) {
	span, ctx := trace.NewSpan(ctx, "Cluster.RefreshTabletReplicationSource")
	defer span.Finish()

	AnnotateSpan(c, span)
	span.Annotate("tablet_alias", topoproto.TabletAliasString(tablet.Tablet.Alias))

	if err := c.topoRWPool.Acquire(ctx); err != nil {
		return nil, fmt.Errorf("RefreshTabletReplicationSource(%v) failed to acquire topoRWPool: %w", topoproto.TabletAliasString(tablet.Tablet.Alias), err)
	}
	defer c.topoRWPool.Release()

	resp, err := c.Vtctld.ReparentTablet(ctx, &vtctldatapb.ReparentTabletRequest{Tablet: tablet.Tablet.Alias})
	if err != nil {
		return nil, err
	}

	return &vtadminpb.RefreshTabletReplicationSourceResponse{
		Keyspace: resp.Keyspace,
		Shard:    resp.Shard,
		Primary:  resp.Primary,
		Cluster:  c.ToProto(),
	}, nil
}

// ReloadSchemas reloads schemas in one or more keyspaces, shards, or tablets
// in the cluster, depending on the request parameters.

View on GitHub (pinned to 01a25a7d17)

Solutions

1. Retry after concurrent topo write operations complete, with a fresh context
2. Serialize ReparentTablet calls client-side
3. Increase topo_rw_pool_size in the vtadmin cluster config
4. Verify no stuck operations are holding pool slots (check in-flight failovers)

Example fix

// before
ctx, cancel := context.WithTimeout(parentCtx, 500*time.Millisecond)
_, err := c.RefreshTabletReplicationSource(ctx, tablet)
// after
ctx, cancel := context.WithTimeout(parentCtx, 30*time.Second)
_, err := c.RefreshTabletReplicationSource(ctx, tablet)

if ctx.Err() != nil {
    return ctx.Err()
}

_, err := c.RefreshTabletReplicationSource(ctx, tablet)
if err != nil && strings.Contains(err.Error(), "failed to acquire topoRWPool") {
    // wait for in-flight reparents, then retry with fresh context
}

When it happens

Trigger: Calling RefreshTabletReplicationSource while other topo writes (failovers, ReparentTablet, external reparent promotions) hold the limited topoRWPool slots, or with a ctx already cancelled.

Common situations: Repointing tablets during a failover storm when several concurrent reparent operations run; scripted tablet re-parenting loops; long-running EmergencyFailoverShard/PlannedFailoverShard holding the RW pool.

Related errors

• PlannedFailoverShard(%s/%s): failed to acquire failoverPool:
• RefreshState(%v) failed to acquire topoReadPool: %w
• TabletExternallyPromoted(%s): failed to acquire topoRWPool:
• invalid choice for enum
• value must be either a float64 (interpreted as seconds) or a

AI-assisted analysis of vitessio/vitess@01a25a7d17 (2026-09-01). Data as JSON: /api/errors/aafa8bd9def7cdbf. Report an issue: GitHub.