{"record":{"id":"6e48d9fc619d8c55","repo":"thanos-io/thanos","slug":"failed-to-wait-for-turn","errorCode":null,"errorMessage":"failed to wait for turn","messagePattern":"failed to wait for turn","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"pkg/query/querier.go","lineNumber":326,"sourceCode":"\t\tctx = fanout.NewContext(ctx, tracker)\n\t}\n\tctx, cancel := context.WithTimeout(ctx, q.selectTimeout)\n\tspan, ctx := tracing.StartSpan(ctx, \"querier_select\", opentracing.Tags{\n\t\t\"minTime\":  hints.Start,\n\t\t\"maxTime\":  hints.End,\n\t\t\"matchers\": \"{\" + strings.Join(matchers, \",\") + \"}\",\n\t})\n\n\tpromise := make(chan storage.SeriesSet, 1)\n\tgo func() {\n\t\tdefer close(promise)\n\n\t\tvar err error\n\t\ttracing.DoInSpan(ctx, \"querier_select_gate_ismyturn\", func(ctx context.Context) {\n\t\t\terr = q.selectGate.Start(ctx)\n\t\t})\n\t\tif err != nil {\n\t\t\tpromise <- storage.ErrSeriesSet(errors.Wrap(err, \"failed to wait for turn\"))\n\t\t\treturn\n\t\t}\n\t\tdefer q.selectGate.Done()\n\n\t\tspan, ctx := tracing.StartSpan(ctx, \"querier_select_select_fn\")\n\t\tdefer span.Finish()\n\n\t\tset, stats, err := q.selectFn(ctx, hints, ms...)\n\t\tif err != nil {\n\t\t\tpromise <- storage.ErrSeriesSet(err)\n\t\t\treturn\n\t\t}\n\t\tq.seriesStatsReporter(stats)\n\n\t\tpromise <- set\n\t}()\n\n\treturn &lazySeriesSet{create: func() (storage.SeriesSet, bool) {","sourceCodeStart":308,"sourceCodeEnd":344,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/pkg/query/querier.go#L308-L344","documentation":"The querier uses a selectGate (concurrency limiter) before running a store Select. Start(ctx) blocks until it is this query's turn; if the context is cancelled or the gate errors while waiting, the select promise resolves with a series set carrying 'failed to wait for turn'.","triggerScenarios":"Calling a PromQL query whose Select hits the querier's selectGate while ctx is cancelled/times out, or the gate's semaphore errors under heavy load.","commonSituations":"Under heavy query concurrency when the concurrency gate queue is long, clients time out and cancel contexts; load balancer timeouts killing long-queued queries.","solutions":["Increase the query concurrency limit (query.concurrency) or reduce concurrent query load","Check/cancel-timeout settings on the client side; ensure ctx deadline exceeds expected queue wait","Investigate slow stores causing long gate queues","Retry the query when load subsides"],"exampleFix":"// before: default low concurrency under heavy load\n// after: raise gate limit\n# prometheus/thanos query flag\n--query.concurrency=20","handlingStrategy":"retry","validationCode":"// ensure ctx has enough budget before Select\nif ctx.Err() != nil || remainingDeadline(ctx) < minQueryBudget {\n    return errors.New(\"insufficient context deadline for query gate\")\n}","typeGuard":null,"tryCatchPattern":"ss := q.Select(ctx, hints, matchers...)\nfor ss.Next() {}\nif err := ss.Err(); err != nil {\n    if strings.Contains(err.Error(), \"failed to wait for turn\") {\n        return retryWithBackoff(ctx, 3) // retry when load subsides\n    }\n    return err\n}","preventionTips":["Raise --query.concurrency to match expected load","Set client timeouts larger than worst-case gate queue wait","Add load-shedding / query-queue limits in the frontend"],"tags":["thanos","query","concurrency","context-cancelled"],"backgroundTag":"request-timeout","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}