{"record":{"id":"93c207cc68e81aa6","repo":"vitessio/vitess","slug":"findworkflows-keyspaces-v-opts-v-failed-t","errorCode":null,"errorMessage":"findWorkflows(keyspaces = %v, opts = %+v) failed to acquire topoReadPool: %w","messagePattern":"findWorkflows\\(keyspaces = (.+?), opts = %\\+v\\) failed to acquire topoReadPool: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"go/vt/vtadmin/cluster/cluster.go","lineNumber":735,"sourceCode":"\treturn c.findWorkflows(ctx, keyspaces, opts)\n}\n\nfunc (c *Cluster) findWorkflows(ctx context.Context, keyspaces []string, opts FindWorkflowsOptions) (*vtadminpb.ClusterWorkflows, error) {\n\tif opts.Filter == nil {\n\t\topts.Filter = func(_ *vtadminpb.Workflow) bool { return true }\n\t}\n\n\tif opts.IgnoreKeyspaces == nil {\n\t\topts.IgnoreKeyspaces = sets.New[string]()\n\t}\n\n\tif len(keyspaces) == 0 {\n\t\tspan, ctx := trace.NewSpan(ctx, \"Cluster.GetKeyspaces\")\n\t\tAnnotateSpan(c, span)\n\n\t\tif err := c.topoReadPool.Acquire(ctx); err != nil {\n\t\t\tspan.Finish()\n\t\t\treturn nil, fmt.Errorf(\"findWorkflows(keyspaces = %v, opts = %+v) failed to acquire topoReadPool: %w\", keyspaces, opts, err)\n\t\t}\n\n\t\tresp, err := c.Vtctld.GetKeyspaces(ctx, &vtctldatapb.GetKeyspacesRequest{})\n\t\tc.topoReadPool.Release()\n\n\t\tif err != nil {\n\t\t\tspan.Finish()\n\t\t\treturn nil, fmt.Errorf(\"GetKeyspaces(cluster = %s) failed: %w\", c.ID, err)\n\t\t}\n\n\t\tfor _, ks := range resp.Keyspaces {\n\t\t\tkeyspaces = append(keyspaces, ks.Name)\n\t\t}\n\n\t\tspan.Finish()\n\t} else if opts.IgnoreKeyspaces.Len() > 0 {\n\t\tlog.Warn(fmt.Sprintf(\"Cluster.findWorkflows: IgnoreKeyspaces was set, but Keyspaces was not empty; ignoring IgnoreKeyspaces in favor of explicitly checking everything in Keyspaces: (%s)\", strings.Join(keyspaces, \", \")))\n\t\topts.IgnoreKeyspaces = sets.New[string]()","sourceCodeStart":717,"sourceCodeEnd":753,"githubUrl":"https://github.com/vitessio/vitess/blob/01a25a7d176f94613b8d59d799f438380a8760e4/go/vt/vtadmin/cluster/cluster.go#L717-L753","documentation":"findWorkflows enumerates all keyspaces (when none are supplied) to discover workflows. It must acquire the read-only topology pool before calling GetKeyspaces; if Acquire(ctx) fails while the pool is exhausted and the context is cancelled/expired, this error is returned (and the span is finished). Reached via FindWorkflows, GetWorkflow, and GetWorkflows.","triggerScenarios":"Calling FindWorkflows/GetWorkflows with no keyspace filter while topoReadPool is saturated; ctx timeout or cancellation during the Acquire wait.","commonSituations":"Cluster-wide workflow dashboards issuing many unfiltered GetWorkflows calls; concurrent schema-tracked-workflows refresh jobs; read pool already contended by FindAllShardsInKeyspace callers.","solutions":["Retry with a longer context deadline","Pass explicit keyspaces to FindWorkflows/GetWorkflows to avoid the unfiltered keyspace enumeration, or reduce concurrent callers","Cache keyspace/workflow lists briefly to cut read-pool pressure","Increase topoReadPool capacity if contention is routine"],"exampleFix":"// before\nworkflows, err := cluster.GetWorkflows(ctx, nil) // enumerates all keyspaces via read pool\n// after\nworkflows, err := cluster.GetWorkflows(ctx, []string{\"commerce\", \"customer\"}) // scoped, less contention","handlingStrategy":"retry","validationCode":"if err := ctx.Err(); err != nil {\n    return fmt.Errorf(\"context already cancelled: %w\", err)\n}","typeGuard":null,"tryCatchPattern":"workflows, err := cluster.GetWorkflows(ctx, nil)\nif err != nil && strings.Contains(err.Error(), \"findWorkflows\") && strings.Contains(err.Error(), \"failed to acquire topoReadPool\") {\n    retryCtx, cancel := context.WithTimeout(context.Background(), 30*time.Second)\n    defer cancel()\n    workflows, err = cluster.GetWorkflows(retryCtx, nil)\n}","preventionTips":["Pass explicit keyspace lists instead of nil to reduce enumeration load","Rate-limit cluster-wide workflow dashboards and refresh jobs","Use bounded concurrency and generous deadlines for read-heavy paths","Increase topoReadPool capacity if contention is a regular occurrence"],"tags":["resource-pool","timeout","concurrency","workflows","vtadmin"],"backgroundTag":"pool-acquisition-failed","analyzedSha":"01a25a7d176f94613b8d59d799f438380a8760e4","analyzedAt":"2026-09-01T17:28:30.605Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}