{"record":{"id":"467b99121370dda7","repo":"vitessio/vitess","slug":"getschema-cluster-s-keyspace-s-tablet-s","errorCode":null,"errorMessage":"GetSchema(cluster = %s, keyspace = %s, tablet = %s) failed to acquire schemaReadPool: %w","messagePattern":"GetSchema\\(cluster = (.+?), keyspace = (.+?), tablet = (.+?)\\) failed to acquire schemaReadPool: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"go/vt/vtadmin/cluster/cluster.go","lineNumber":1694,"sourceCode":"\tfor _, tablet := range tablets {\n\t\twg.Add(1)\n\n\t\tgo func(tablet *vtadminpb.Tablet, sizesOnly bool) {\n\t\t\tdefer wg.Done()\n\n\t\t\tspan, ctx := trace.NewSpan(ctx, \"Vtctld.GetSchema\")\n\t\t\tdefer span.Finish()\n\t\t\treq := opts.BaseRequest.CloneVT()\n\t\t\treq.TableSizesOnly = sizesOnly\n\t\t\treq.TabletAlias = tablet.Tablet.Alias\n\n\t\t\tAnnotateSpan(c, span)\n\t\t\tannotateGetSchemaRequest(req, span)\n\t\t\tspan.Annotate(\"keyspace\", keyspace)\n\t\t\tspan.Annotate(\"shard\", tablet.Tablet.Shard)\n\n\t\t\tif err := c.schemaReadPool.Acquire(ctx); err != nil {\n\t\t\t\terr = fmt.Errorf(\"GetSchema(cluster = %s, keyspace = %s, tablet = %s) failed to acquire schemaReadPool: %w\", c.ID, keyspace, tablet.Tablet.Alias, err)\n\t\t\t\trec.RecordError(err)\n\t\t\t\treturn\n\t\t\t}\n\n\t\t\tresp, err := c.Vtctld.GetSchema(ctx, req)\n\t\t\tc.schemaReadPool.Release()\n\n\t\t\tif err != nil {\n\t\t\t\terr = fmt.Errorf(\"GetSchema(cluster = %s, keyspace = %s, tablet = %s) failed: %w\", c.ID, keyspace, tablet.Tablet.Alias, err)\n\t\t\t\trec.RecordError(err)\n\n\t\t\t\treturn\n\t\t\t}\n\n\t\t\tif resp == nil || resp.Schema == nil {\n\t\t\t\treturn\n\t\t\t}\n","sourceCodeStart":1676,"sourceCodeEnd":1712,"githubUrl":"https://github.com/vitessio/vitess/blob/01a25a7d176f94613b8d59d799f438380a8760e4/go/vt/vtadmin/cluster/cluster.go#L1676-L1712","documentation":"For each tablet selected during GetSchema, vtadmin acquires schemaReadPool before issuing Vtctld.GetSchema; an Acquire failure (ctx cancelled/deadline while waiting for a slot) is wrapped with cluster, keyspace, and tablet alias and recorded into the error recorder. It is a resource-limiting failure, not a schema error.","triggerScenarios":"Large GetSchema fan-outs across many tablets holding all schemaReadPool slots, with the caller's ctx cancelled or deadline exceeded while a goroutine waits to query a specific tablet.","commonSituations":"Schema refresh across hundreds of tablets with a small pool; slow/unresponsive tablets making GetSchema calls hold slots for a long time; UI request timeouts cancelling ctx mid-wait.","solutions":["Increase schemaReadPool size in vtadmin config","Check the wrapped cause: DeadlineExceeded indicates sustained pool saturation","Reduce the tablet set queried (use narrower opts/tablet filters)","Investigate slow tablets (overloaded mysqld, network) that hold pool slots"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// Go: check remaining context budget before the fan-out\ndeadline, ok := ctx.Deadline()\nif !ok || time.Until(deadline) < poolWaitBudget {\n\treturn fmt.Errorf(\"insufficient deadline for schema fan-out\")\n}","typeGuard":null,"tryCatchPattern":"schema, err := c.GetSchema(ctx, req)\nif err != nil {\n\tif strings.Contains(err.Error(), \"failed to acquire schemaReadPool\") {\n\t\t// pool exhaustion: retry with backoff or raise pool size\n\t}\n\treturn err\n}","preventionTips":["Size schemaReadPool to the expected tablet fan-out width","Replace or quarantine persistently slow tablets","Limit per-request tablet counts","Monitor schemaReadPool saturation metrics"],"tags":["vtadmin","semaphore","schema","concurrency"],"backgroundTag":"read-pool-exhausted","analyzedSha":"01a25a7d176f94613b8d59d799f438380a8760e4","analyzedAt":"2026-09-01T17:28:30.605Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}