{"record":{"id":"1a6fbad2067964aa","repo":"vitessio/vitess","slug":"getkeyspaces-failed-to-acquire-toporeadpool-w","errorCode":null,"errorMessage":"GetKeyspaces() failed to acquire topoReadPool: %w","messagePattern":"GetKeyspaces\\(\\) failed to acquire topoReadPool: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"go/vt/vtadmin/cluster/cluster.go","lineNumber":1214,"sourceCode":"\t\treturn nil, err\n\t}\n\n\treturn &vtadminpb.Keyspace{\n\t\tCluster:  c.ToProto(),\n\t\tKeyspace: resp.Keyspace,\n\t\tShards:   shards,\n\t}, nil\n}\n\n// GetKeyspaces returns all keyspaces, with their shard maps, in the cluster.\nfunc (c *Cluster) GetKeyspaces(ctx context.Context) ([]*vtadminpb.Keyspace, error) {\n\tspan, ctx := trace.NewSpan(ctx, \"Cluster.GetKeyspaces\")\n\tdefer span.Finish()\n\n\tAnnotateSpan(c, span)\n\n\tif err := c.topoReadPool.Acquire(ctx); err != nil {\n\t\treturn nil, fmt.Errorf(\"GetKeyspaces() failed to acquire topoReadPool: %w\", err)\n\t}\n\n\tresp, err := c.Vtctld.GetKeyspaces(ctx, &vtctldatapb.GetKeyspacesRequest{})\n\tc.topoReadPool.Release()\n\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\n\tvar (\n\t\tm         sync.Mutex\n\t\twg        sync.WaitGroup\n\t\trec       concurrency.AllErrorRecorder\n\t\tkeyspaces = make([]*vtadminpb.Keyspace, len(resp.Keyspaces))\n\t)\n\n\tfor i, ks := range resp.Keyspaces {\n\t\twg.Add(1)","sourceCodeStart":1196,"sourceCodeEnd":1232,"githubUrl":"https://github.com/vitessio/vitess/blob/01a25a7d176f94613b8d59d799f438380a8760e4/go/vt/vtadmin/cluster/cluster.go#L1196-L1232","documentation":"Cluster.GetKeyspaces acquires topoReadPool before calling Vtctld.GetKeyspaces; failure to acquire means ctx cancellation/deadline while waiting for a free slot. This variant returns the error directly to the caller (unlike error 866 which records it into an error recorder).","triggerScenarios":"Calling cluster.GetKeyspaces(ctx) (e.g. via getShardSets) with all topoReadPool slots held and ctx cancelled before a slot becomes available.","commonSituations":"Many concurrent vtadmin schema/keyspace operations saturating the topo read pool; slow topo under load; caller-side timeouts firing while requests queue on the semaphore.","solutions":["Inspect the wrapped cause to distinguish cancellation vs deadline-exceeded","Increase topoReadPool size in vtadmin config","Stagger or rate-limit bulk keyspace/shard operations","Check vtctld/topo latency that keeps slots occupied"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// Go: ensure ctx still alive and caller deadline sane before the call\nif ctx.Err() != nil {\n\treturn ctx.Err()\n}","typeGuard":null,"tryCatchPattern":"resp, err := c.GetKeyspaces(ctx)\nif err != nil && strings.Contains(err.Error(), \"failed to acquire topoReadPool\") {\n\t// exponential backoff retry; or enlarge pool\n}","preventionTips":["Increase topoReadPool size in vtadmin config","Avoid concurrent getShardSets storms; serialize or batch","Set realistic deadlines based on topo latency","Alert on repeated pool-acquisition failures"],"tags":["vtadmin","semaphore","concurrency","timeout"],"backgroundTag":"read-pool-exhausted","analyzedSha":"01a25a7d176f94613b8d59d799f438380a8760e4","analyzedAt":"2026-09-01T17:28:30.605Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}