vitessio/vitess · warning

GetWorkflows(keyspace = %s, active_only = %v) failed to acqu

Error message

GetWorkflows(keyspace = %s, active_only = %v) failed to acquire workflowReadPool: %w

What it means

GetWorkflows runs one goroutine per keyspace, each gated by a bounded semaphore (workflowReadPool). If the semaphore slot cannot be acquired before the request context is done, this error is recorded and the per-keyspace lookup is skipped. It signals concurrency-limit/context pressure, not a vtctld failure.

Source

Thrown at go/vt/vtadmin/cluster/cluster.go:793

			log.Info("Cluster.findWorkflows: ignoring keyspace " + ks)

			continue
		}

		wg.Add(1)

		go func(ks string) {
			defer wg.Done()

			span, ctx := trace.NewSpan(ctx, "Cluster.GetWorkflowsForKeyspace")
			defer span.Finish()

			AnnotateSpan(c, span)
			span.Annotate("keyspace", ks)
			span.Annotate("active_only", opts.ActiveOnly)

			if err := c.workflowReadPool.Acquire(ctx); err != nil {
				err = fmt.Errorf("GetWorkflows(keyspace = %s, active_only = %v) failed to acquire workflowReadPool: %w", ks, opts.ActiveOnly, err)
				rec.RecordError(err)

				return
			}

			resp, err := c.Vtctld.GetWorkflows(ctx, &vtctldatapb.GetWorkflowsRequest{
				Keyspace:    ks,
				ActiveOnly:  opts.ActiveOnly,
				IncludeLogs: true,
			})
			c.workflowReadPool.Release()

			if err != nil {
				err = fmt.Errorf("GetWorkflows(keyspace = %s, active_only = %v) failed: %w", ks, opts.ActiveOnly, err)
				rec.RecordError(err)

				return
			}

View on GitHub (pinned to 01a25a7d17)

Solutions

1. Retry with a longer request deadline
2. Increase the workflowReadPool size in the cluster configuration
3. Reduce concurrent vtadmin workflow requests
4. Check whether Acquire ever returns promptly; if ctx deadlines are too short, raise them client-side

Example fix

// before
if err := c.workflowReadPool.Acquire(ctx); err != nil { ... }
// after (caller): give the request more headroom
ctx, cancel := context.WithTimeout(ctx, 60*time.Second)
defer cancel()

res, err := cluster.GetWorkflows(ctx, ks, opts)
if err != nil && strings.Contains(err.Error(), "failed to acquire workflowReadPool") {
    // ctx deadline hit while waiting; retry with more time or fewer keyspacees
}

When it happens

Trigger: Invoking GetWorkflows (or FindWorkflows which calls it) with many keyspaces while workflowReadPool is saturated and the caller's ctx is canceled or times out while blocked on c.workflowReadPool.Acquire(ctx).

Common situations: Enumerating workflows across hundreds of keyspaces with a small read pool; client-side HTTP timeout shorter than the pool wait; bursty concurrent vtadmin requests exhausting the pool.

Related errors

• GetBackups(%s/%s) failed to acquire backupReadPool: %w
• GetCellInfo(%s) failed to acquire topoReadPool: %w
• GetKeyspace(%s) failed to acquire topoReadPool: %w
• GetKeyspaces() failed to acquire topoReadPool: %w
• invalid choice for enum

AI-assisted analysis of vitessio/vitess@01a25a7d17 (2026-09-01). Data as JSON: /api/errors/538fba420697bfea. Report an issue: GitHub.