{"record":{"id":"538fba420697bfea","repo":"vitessio/vitess","slug":"getworkflows-keyspace-s-active-only-v-fail","errorCode":null,"errorMessage":"GetWorkflows(keyspace = %s, active_only = %v) failed to acquire workflowReadPool: %w","messagePattern":"GetWorkflows\\(keyspace = (.+?), active_only = (.+?)\\) failed to acquire workflowReadPool: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"go/vt/vtadmin/cluster/cluster.go","lineNumber":793,"sourceCode":"\t\t\tlog.Info(\"Cluster.findWorkflows: ignoring keyspace \" + ks)\n\n\t\t\tcontinue\n\t\t}\n\n\t\twg.Add(1)\n\n\t\tgo func(ks string) {\n\t\t\tdefer wg.Done()\n\n\t\t\tspan, ctx := trace.NewSpan(ctx, \"Cluster.GetWorkflowsForKeyspace\")\n\t\t\tdefer span.Finish()\n\n\t\t\tAnnotateSpan(c, span)\n\t\t\tspan.Annotate(\"keyspace\", ks)\n\t\t\tspan.Annotate(\"active_only\", opts.ActiveOnly)\n\n\t\t\tif err := c.workflowReadPool.Acquire(ctx); err != nil {\n\t\t\t\terr = fmt.Errorf(\"GetWorkflows(keyspace = %s, active_only = %v) failed to acquire workflowReadPool: %w\", ks, opts.ActiveOnly, err)\n\t\t\t\trec.RecordError(err)\n\n\t\t\t\treturn\n\t\t\t}\n\n\t\t\tresp, err := c.Vtctld.GetWorkflows(ctx, &vtctldatapb.GetWorkflowsRequest{\n\t\t\t\tKeyspace:    ks,\n\t\t\t\tActiveOnly:  opts.ActiveOnly,\n\t\t\t\tIncludeLogs: true,\n\t\t\t})\n\t\t\tc.workflowReadPool.Release()\n\n\t\t\tif err != nil {\n\t\t\t\terr = fmt.Errorf(\"GetWorkflows(keyspace = %s, active_only = %v) failed: %w\", ks, opts.ActiveOnly, err)\n\t\t\t\trec.RecordError(err)\n\n\t\t\t\treturn\n\t\t\t}","sourceCodeStart":775,"sourceCodeEnd":811,"githubUrl":"https://github.com/vitessio/vitess/blob/01a25a7d176f94613b8d59d799f438380a8760e4/go/vt/vtadmin/cluster/cluster.go#L775-L811","documentation":"GetWorkflows runs one goroutine per keyspace, each gated by a bounded semaphore (workflowReadPool). If the semaphore slot cannot be acquired before the request context is done, this error is recorded and the per-keyspace lookup is skipped. It signals concurrency-limit/context pressure, not a vtctld failure.","triggerScenarios":"Invoking GetWorkflows (or FindWorkflows which calls it) with many keyspaces while workflowReadPool is saturated and the caller's ctx is canceled or times out while blocked on c.workflowReadPool.Acquire(ctx).","commonSituations":"Enumerating workflows across hundreds of keyspaces with a small read pool; client-side HTTP timeout shorter than the pool wait; bursty concurrent vtadmin requests exhausting the pool.","solutions":["Retry with a longer request deadline","Increase the workflowReadPool size in the cluster configuration","Reduce concurrent vtadmin workflow requests","Check whether Acquire ever returns promptly; if ctx deadlines are too short, raise them client-side"],"exampleFix":"// before\nif err := c.workflowReadPool.Acquire(ctx); err != nil { ... }\n// after (caller): give the request more headroom\nctx, cancel := context.WithTimeout(ctx, 60*time.Second)\ndefer cancel()","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"res, err := cluster.GetWorkflows(ctx, ks, opts)\nif err != nil && strings.Contains(err.Error(), \"failed to acquire workflowReadPool\") {\n    // ctx deadline hit while waiting; retry with more time or fewer keyspacees\n}","preventionTips":["Size workflowReadPool for your keyspace count","Avoid very short client deadlines","Throttle concurrent workflow requests","Back off and retry on pool-acquisition failures"],"tags":["vtadmin","semaphore","concurrency","timeout"],"backgroundTag":"semaphore-acquire-cancelled","analyzedSha":"01a25a7d176f94613b8d59d799f438380a8760e4","analyzedAt":"2026-09-01T17:28:30.605Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}