findWorkflows(keyspaces = %v, opts = %+v) failed to acquire

Error message

findWorkflows(keyspaces = %v, opts = %+v) failed to acquire topoReadPool: %w

What it means

findWorkflows enumerates all keyspaces (when none are supplied) to discover workflows. It must acquire the read-only topology pool before calling GetKeyspaces; if Acquire(ctx) fails while the pool is exhausted and the context is cancelled/expired, this error is returned (and the span is finished). Reached via FindWorkflows, GetWorkflow, and GetWorkflows.

Source

Thrown at go/vt/vtadmin/cluster/cluster.go:735

	return c.findWorkflows(ctx, keyspaces, opts)
}

func (c *Cluster) findWorkflows(ctx context.Context, keyspaces []string, opts FindWorkflowsOptions) (*vtadminpb.ClusterWorkflows, error) {
	if opts.Filter == nil {
		opts.Filter = func(_ *vtadminpb.Workflow) bool { return true }
	}

	if opts.IgnoreKeyspaces == nil {
		opts.IgnoreKeyspaces = sets.New[string]()
	}

	if len(keyspaces) == 0 {
		span, ctx := trace.NewSpan(ctx, "Cluster.GetKeyspaces")
		AnnotateSpan(c, span)

		if err := c.topoReadPool.Acquire(ctx); err != nil {
			span.Finish()
			return nil, fmt.Errorf("findWorkflows(keyspaces = %v, opts = %+v) failed to acquire topoReadPool: %w", keyspaces, opts, err)
		}

		resp, err := c.Vtctld.GetKeyspaces(ctx, &vtctldatapb.GetKeyspacesRequest{})
		c.topoReadPool.Release()

		if err != nil {
			span.Finish()
			return nil, fmt.Errorf("GetKeyspaces(cluster = %s) failed: %w", c.ID, err)
		}

		for _, ks := range resp.Keyspaces {
			keyspaces = append(keyspaces, ks.Name)
		}

		span.Finish()
	} else if opts.IgnoreKeyspaces.Len() > 0 {
		log.Warn(fmt.Sprintf("Cluster.findWorkflows: IgnoreKeyspaces was set, but Keyspaces was not empty; ignoring IgnoreKeyspaces in favor of explicitly checking everything in Keyspaces: (%s)", strings.Join(keyspaces, ", ")))
		opts.IgnoreKeyspaces = sets.New[string]()

View on GitHub (pinned to 01a25a7d17)

Solutions

Retry with a longer context deadline
Pass explicit keyspaces to FindWorkflows/GetWorkflows to avoid the unfiltered keyspace enumeration, or reduce concurrent callers
Cache keyspace/workflow lists briefly to cut read-pool pressure
Increase topoReadPool capacity if contention is routine

Example fix

// before
workflows, err := cluster.GetWorkflows(ctx, nil) // enumerates all keyspaces via read pool
// after
workflows, err := cluster.GetWorkflows(ctx, []string{"commerce", "customer"}) // scoped, less contention

Defensive patterns

Strategy: retry

Validate before calling

if err := ctx.Err(); err != nil {
    return fmt.Errorf("context already cancelled: %w", err)
}

Try / catch

workflows, err := cluster.GetWorkflows(ctx, nil)
if err != nil && strings.Contains(err.Error(), "findWorkflows") && strings.Contains(err.Error(), "failed to acquire topoReadPool") {
    retryCtx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
    defer cancel()
    workflows, err = cluster.GetWorkflows(retryCtx, nil)
}

Prevention

Pass explicit keyspace lists instead of nil to reduce enumeration load
Rate-limit cluster-wide workflow dashboards and refresh jobs
Use bounded concurrency and generous deadlines for read-heavy paths
Increase topoReadPool capacity if contention is a regular occurrence

When it happens

Trigger: Calling FindWorkflows/GetWorkflows with no keyspace filter while topoReadPool is saturated; ctx timeout or cancellation during the Acquire wait.

Common situations: Cluster-wide workflow dashboards issuing many unfiltered GetWorkflows calls; concurrent schema-tracked-workflows refresh jobs; read pool already contended by FindAllShardsInKeyspace callers.

Related errors

AI-assisted analysis of vitessio/vitess@01a25a7d17 (2026-09-01). Data as JSON: /api/errors/93c207cc68e81aa6. Report an issue: GitHub.