{"record":{"id":"28bcee5c545e6226","repo":"vitessio/vitess","slug":"findallshardsinkeyspace-s-failed-to-acquire-topo","errorCode":null,"errorMessage":"FindAllShardsInKeyspace(%s) failed to acquire topoReadPool: %w","messagePattern":"FindAllShardsInKeyspace\\((.+?)\\) failed to acquire topoReadPool: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"go/vt/vtadmin/cluster/cluster.go","lineNumber":623,"sourceCode":"\t// outside this package.\n\tskipPool bool\n}\n\n// FindAllShardsInKeyspace proxies a FindAllShardsInKeyspace RPC to a cluster's\n// vtctld, unpacking the response struct.\n//\n// It can also optionally ensure the vtctldclient has a valid connection before\n// making the RPC call.\nfunc (c *Cluster) FindAllShardsInKeyspace(ctx context.Context, keyspace string, opts FindAllShardsInKeyspaceOptions) (map[string]*vtctldatapb.Shard, error) {\n\tspan, ctx := trace.NewSpan(ctx, \"Cluster.FindAllShardsInKeyspace\")\n\tdefer span.Finish()\n\n\tAnnotateSpan(c, span)\n\tspan.Annotate(\"keyspace\", keyspace)\n\n\tif !opts.skipPool {\n\t\tif err := c.topoReadPool.Acquire(ctx); err != nil {\n\t\t\treturn nil, fmt.Errorf(\"FindAllShardsInKeyspace(%s) failed to acquire topoReadPool: %w\", keyspace, err)\n\t\t}\n\t\tdefer c.topoReadPool.Release()\n\t}\n\n\tresp, err := c.Vtctld.FindAllShardsInKeyspace(ctx, &vtctldatapb.FindAllShardsInKeyspaceRequest{\n\t\tKeyspace: keyspace,\n\t})\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"FindAllShardsInKeyspace(cluster = %s, keyspace = %s) failed: %w\", c.ID, keyspace, err)\n\t}\n\n\treturn resp.Shards, nil\n}\n\n// FindTablet returns the first tablet in a given cluster that satisfies the filter function.\nfunc (c *Cluster) FindTablet(ctx context.Context, filter func(*vtadminpb.Tablet) bool) (*vtadminpb.Tablet, error) {\n\tspan, ctx := trace.NewSpan(ctx, \"Cluster.FindTablet\")\n\tdefer span.Finish()","sourceCodeStart":605,"sourceCodeEnd":641,"githubUrl":"https://github.com/vitessio/vitess/blob/01a25a7d176f94613b8d59d799f438380a8760e4/go/vt/vtadmin/cluster/cluster.go#L605-L641","documentation":"FindAllShardsInKeyspace acquires the read-only topology pool before querying vtctld, unless opts.skipPool is set (used for re-entrant calls already holding a slot). If Acquire(ctx) fails — read pool exhausted and the context cancelled/timed out waiting — this error is wrapped with the keyspace. The read pool has more capacity than the RW pool but is still bounded.","triggerScenarios":"Heavy concurrent reads (GetWorkflows, GetKeyspace, schema queries) exhausting topoReadPool; ctx deadline exceeded or cancelled while blocked in Acquire; opts.skipPool false when the caller is not already holding a slot.","commonSituations":"Dashboards polling many keyspaces concurrently; vtadmin web UI burst on page load; long schema-refresh operations holding read slots.","solutions":["Retry the call with a longer context deadline","Reduce the number of concurrent keyspace-wide reads (cache results, batch keyspace lists)","If calling from within a function that already holds the read pool, use the opts.skipPool path","Increase topoReadPool capacity if the workload legitimately needs it"],"exampleFix":"// before\nfor _, ks := range allKeyspaces {\n    go cluster.FindAllShardsInKeyspace(ctx, ks, nil) // 500 keyspaces, read pool exhausted\n}\n// after\ng, gctx := errgroup.WithContext(ctx)\ng.SetLimit(8)\nfor _, ks := range allKeyspaces {\n    ks := ks\n    g.Go(func() error { _, err := cluster.FindAllShardsInKeyspace(gctx, ks, nil); return err })\n}","handlingStrategy":"retry","validationCode":"if keyspace == \"\" {\n    return fmt.Errorf(\"keyspace name is required\")\n}\nif err := ctx.Err(); err != nil {\n    return fmt.Errorf(\"context already cancelled: %w\", err)\n}","typeGuard":null,"tryCatchPattern":"shards, err := cluster.FindAllShardsInKeyspace(ctx, ks, nil)\nif err != nil && strings.Contains(err.Error(), \"failed to acquire topoReadPool\") {\n    retryCtx, cancel := context.WithTimeout(context.Background(), 30*time.Second)\n    defer cancel()\n    shards, err = cluster.FindAllShardsInKeyspace(retryCtx, ks, nil)\n}","preventionTips":["Limit concurrency of keyspace-wide reads with an errgroup SetLimit","Pass opts with skipPool only when already holding the read pool","Cache shard listings for short periods on read-heavy dashboards"],"tags":["resource-pool","timeout","concurrency","vtadmin"],"backgroundTag":"pool-acquisition-failed","analyzedSha":"01a25a7d176f94613b8d59d799f438380a8760e4","analyzedAt":"2026-09-01T17:28:30.605Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}