vitessio/vitess · warning

GetBackups(%s/%s) failed to acquire backupReadPool: %w

Error message

GetBackups(%s/%s) failed to acquire backupReadPool: %w

What it means

GetBackups fans out a goroutine per keyspace/shard, each guarded by the backupReadPool semaphore. Failing to Acquire a slot before the RPC means the context ended (deadline/cancel) while waiting, so that shard's backup listing is skipped and recorded as this error.

Source

Thrown at go/vt/vtadmin/cluster/cluster.go:882

		clusterProto = c.ToProto()
	)

	for ks, shardSet := range shardsByKeyspace {
		for _, shard := range sets.List(shardSet) {
			wg.Add(1)

			go func(keyspace, shard string) {
				defer wg.Done()

				span, ctx := trace.NewSpan(ctx, "Cluster.getBackupsForShard")
				defer span.Finish()

				AnnotateSpan(c, span)
				span.Annotate("keyspace", keyspace)
				span.Annotate("shard", shard)

				if err := c.backupReadPool.Acquire(ctx); err != nil {
					rec.RecordError(fmt.Errorf("GetBackups(%s/%s) failed to acquire backupReadPool: %w", keyspace, shard, err))
					return
				}

				resp, err := c.Vtctld.GetBackups(ctx, &vtctldatapb.GetBackupsRequest{
					Keyspace:      keyspace,
					Shard:         shard,
					Limit:         req.RequestOptions.Limit,
					Detailed:      req.RequestOptions.Detailed,
					DetailedLimit: req.RequestOptions.DetailedLimit,
				})
				c.backupReadPool.Release()

				if err != nil {
					rec.RecordError(fmt.Errorf("GetBackups(%s/%s): %w", keyspace, shard, err))
					return
				}

				shardBackups := make([]*vtadminpb.ClusterBackup, len(resp.Backups))

View on GitHub (pinned to 01a25a7d17)

Solutions

1. Retry with a longer timeout
2. Increase backupReadPool capacity in cluster config
3. Throttle concurrent backup-listing requests
4. Verify no request hangs holding pool slots (e.g. slow vtctld)

backups, err := cluster.GetBackups(ctx, req)
if err != nil && strings.Contains(err.Error(), "backupReadPool") {
    // pool/ctx pressure: back off and retry with longer timeout
}

When it happens

Trigger: Calling GetBackups across many shards while backupReadPool is full and ctx is canceled or times out waiting on c.backupReadPool.Acquire(ctx).

Common situations: Clusters with very large shard counts relative to the pool size; short client timeouts; many simultaneous vtadmin dashboard/backup requests contending for slots.

Related errors

• GetWorkflows(keyspace = %s, active_only = %v) failed to acqu
• GetCellInfo(%s) failed to acquire topoReadPool: %w
• GetKeyspace(%s) failed to acquire topoReadPool: %w
• GetKeyspaces() failed to acquire topoReadPool: %w
• invalid choice for enum

AI-assisted analysis of vitessio/vitess@01a25a7d17 (2026-09-01). Data as JSON: /api/errors/a22750887768b0cf. Report an issue: GitHub.