{"record":{"id":"a22750887768b0cf","repo":"vitessio/vitess","slug":"getbackups-s-s-failed-to-acquire-backupreadpool","errorCode":null,"errorMessage":"GetBackups(%s/%s) failed to acquire backupReadPool: %w","messagePattern":"GetBackups\\((.+?)/(.+?)\\) failed to acquire backupReadPool: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"go/vt/vtadmin/cluster/cluster.go","lineNumber":882,"sourceCode":"\t\tclusterProto = c.ToProto()\n\t)\n\n\tfor ks, shardSet := range shardsByKeyspace {\n\t\tfor _, shard := range sets.List(shardSet) {\n\t\t\twg.Add(1)\n\n\t\t\tgo func(keyspace, shard string) {\n\t\t\t\tdefer wg.Done()\n\n\t\t\t\tspan, ctx := trace.NewSpan(ctx, \"Cluster.getBackupsForShard\")\n\t\t\t\tdefer span.Finish()\n\n\t\t\t\tAnnotateSpan(c, span)\n\t\t\t\tspan.Annotate(\"keyspace\", keyspace)\n\t\t\t\tspan.Annotate(\"shard\", shard)\n\n\t\t\t\tif err := c.backupReadPool.Acquire(ctx); err != nil {\n\t\t\t\t\trec.RecordError(fmt.Errorf(\"GetBackups(%s/%s) failed to acquire backupReadPool: %w\", keyspace, shard, err))\n\t\t\t\t\treturn\n\t\t\t\t}\n\n\t\t\t\tresp, err := c.Vtctld.GetBackups(ctx, &vtctldatapb.GetBackupsRequest{\n\t\t\t\t\tKeyspace:      keyspace,\n\t\t\t\t\tShard:         shard,\n\t\t\t\t\tLimit:         req.RequestOptions.Limit,\n\t\t\t\t\tDetailed:      req.RequestOptions.Detailed,\n\t\t\t\t\tDetailedLimit: req.RequestOptions.DetailedLimit,\n\t\t\t\t})\n\t\t\t\tc.backupReadPool.Release()\n\n\t\t\t\tif err != nil {\n\t\t\t\t\trec.RecordError(fmt.Errorf(\"GetBackups(%s/%s): %w\", keyspace, shard, err))\n\t\t\t\t\treturn\n\t\t\t\t}\n\n\t\t\t\tshardBackups := make([]*vtadminpb.ClusterBackup, len(resp.Backups))","sourceCodeStart":864,"sourceCodeEnd":900,"githubUrl":"https://github.com/vitessio/vitess/blob/01a25a7d176f94613b8d59d799f438380a8760e4/go/vt/vtadmin/cluster/cluster.go#L864-L900","documentation":"GetBackups fans out a goroutine per keyspace/shard, each guarded by the backupReadPool semaphore. Failing to Acquire a slot before the RPC means the context ended (deadline/cancel) while waiting, so that shard's backup listing is skipped and recorded as this error.","triggerScenarios":"Calling GetBackups across many shards while backupReadPool is full and ctx is canceled or times out waiting on c.backupReadPool.Acquire(ctx).","commonSituations":"Clusters with very large shard counts relative to the pool size; short client timeouts; many simultaneous vtadmin dashboard/backup requests contending for slots.","solutions":["Retry with a longer timeout","Increase backupReadPool capacity in cluster config","Throttle concurrent backup-listing requests","Verify no request hangs holding pool slots (e.g. slow vtctld)"],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"backups, err := cluster.GetBackups(ctx, req)\nif err != nil && strings.Contains(err.Error(), \"backupReadPool\") {\n    // pool/ctx pressure: back off and retry with longer timeout\n}","preventionTips":["Size backupReadPool to shard fan-out","Use longer timeouts for cluster-wide backup sweeps","Stagger backup-listing requests","Watch for slow vtctld calls holding pool slots"],"tags":["vtadmin","semaphore","concurrency","timeout"],"backgroundTag":"semaphore-acquire-cancelled","analyzedSha":"01a25a7d176f94613b8d59d799f438380a8760e4","analyzedAt":"2026-09-01T17:28:30.605Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}