thanos-io/thanos · warning

failed to wait for turn

Error message

failed to wait for turn

What it means

The querier uses a selectGate (concurrency limiter) before running a store Select. Start(ctx) blocks until it is this query's turn; if the context is cancelled or the gate errors while waiting, the select promise resolves with a series set carrying 'failed to wait for turn'.

Solutions

1. Increase the query concurrency limit (query.concurrency) or reduce concurrent query load
2. Check/cancel-timeout settings on the client side; ensure ctx deadline exceeds expected queue wait
3. Investigate slow stores causing long gate queues
4. Retry the query when load subsides

Example fix

// before: default low concurrency under heavy load
// after: raise gate limit
# prometheus/thanos query flag
--query.concurrency=20

// ensure ctx has enough budget before Select
if ctx.Err() != nil || remainingDeadline(ctx) < minQueryBudget {
    return errors.New("insufficient context deadline for query gate")
}

ss := q.Select(ctx, hints, matchers...)
for ss.Next() {}
if err := ss.Err(); err != nil {
    if strings.Contains(err.Error(), "failed to wait for turn") {
        return retryWithBackoff(ctx, 3) // retry when load subsides
    }
    return err
}

When it happens

Trigger: Calling a PromQL query whose Select hits the querier's selectGate while ctx is cancelled/times out, or the gate's semaphore errors under heavy load.

Common situations: Under heavy query concurrency when the concurrency gate queue is long, clients time out and cancel contexts; load balancer timeouts killing long-queued queries.

Understand the failure class

Background: Request timed out: what client-side request timeouts mean across libraries (Request timed out, TIMED_OUT, APITimeoutError) — this error's family across 39 libraries.

Related errors

• unknown partial response strategy
• add thanos opts query params
• unexpected result aggregate type
• no valid chunk found
• got empty chunks

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/6e48d9fc619d8c55. Report an issue: GitHub.

Appendix: source

Thrown at pkg/query/querier.go:326

		ctx = fanout.NewContext(ctx, tracker)
	}
	ctx, cancel := context.WithTimeout(ctx, q.selectTimeout)
	span, ctx := tracing.StartSpan(ctx, "querier_select", opentracing.Tags{
		"minTime":  hints.Start,
		"maxTime":  hints.End,
		"matchers": "{" + strings.Join(matchers, ",") + "}",
	})

	promise := make(chan storage.SeriesSet, 1)
	go func() {
		defer close(promise)

		var err error
		tracing.DoInSpan(ctx, "querier_select_gate_ismyturn", func(ctx context.Context) {
			err = q.selectGate.Start(ctx)
		})
		if err != nil {
			promise <- storage.ErrSeriesSet(errors.Wrap(err, "failed to wait for turn"))
			return
		}
		defer q.selectGate.Done()

		span, ctx := tracing.StartSpan(ctx, "querier_select_select_fn")
		defer span.Finish()

		set, stats, err := q.selectFn(ctx, hints, ms...)
		if err != nil {
			promise <- storage.ErrSeriesSet(err)
			return
		}
		q.seriesStatsReporter(stats)

		promise <- set
	}()

	return &lazySeriesSet{create: func() (storage.SeriesSet, bool) {

View on GitHub (pinned to 35b8b99117)