thanos-io/thanos · warning

failed to wait for turn

Error message

failed to wait for turn

What it means

The store gateway uses a concurrency-limiting query gate before serving Series requests. This error wraps any failure from queryGate.Start() (waiting for a turn to execute) during the gRPC Series handler. Typically the underlying cause is the caller's context being canceled (client disconnect/timeout) or the gate's limiter rejecting entry while shutting down.

Solutions

1. Check the wrapped cause (context canceled vs deadline exceeded) in the error chain to distinguish client disconnects from overload.
2. Increase --store.grpc.series-sample-concurrency (or the query gate limit) or reduce query fan-out to shorten wait times.
3. Raise querier/store timeout values so queries are not killed while queued.
4. Retry idempotent queries on DeadlineExceeded; investigate load if the gate is persistently saturated.

Example fix

// querier side: tolerate transient gate waits
cctx, cancel := context.WithTimeout(ctx, 2*time.Minute)
defer cancel()
series, err := storeClient.Series(cctx, req)
if errors.Is(err, context.DeadlineExceeded) {
    series, err = storeClient.Series(ctx, req) // retry once
}

// before calling, check gate saturation if exposed
if storeGate != nil && storeGate.IsBlocked() { /* wait or back off before issuing the call */ }

err := s.queryGate.Start(ctx)
if err != nil {
    if errors.Is(err, context.Canceled) {
        return status.Error(codes.Canceled, "client canceled while waiting for turn")
    }
    if errors.Is(err, context.DeadlineExceeded) {
        return status.Error(codes.DeadlineExceeded, "timed out waiting for query gate")
    }
    return status.Error(codes.Unavailable, err.Error())
}

When it happens

Trigger: A gRPC Series call arrives while the store's query gate is saturated or the store is shutting down; the client context is canceled before a concurrency slot becomes free, so queryGate.Start(srv.Context()) returns context.Canceled/DeadlineExceeded.

Common situations: Query fan-out over many blocks exceeding the store's configured concurrency; querier timeouts shorter than the gate wait time; client cancellation mid-query; store restart under load.

Understand the failure class

Background: Request timed out: what client-side request timeouts mean across libraries (Request timed out, TIMED_OUT, APITimeoutError) — this error's family across 39 libraries.

Related errors

• building gRPC client
• endpoint
• Not ready
• not ready
• cannot obtain metadata: neither info nor store client found

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/fc4923cbc70c2084. Report an issue: GitHub.

Appendix: source

Thrown at pkg/store/bucket.go:1559

	}

	parts = append(parts, fmt.Sprintf("Range: %d-%d Resolution: %d", currMin, currMax, currRes))

	level.Debug(logger).Log("msg", "Blocks source resolutions", "blocks", len(bs), "Maximum Resolution", maxResolutionMillis, "mint", mint, "maxt", maxt, "lset", lset.String(), "spans", strings.Join(parts, "\n"))
}

// Series implements the storepb.StoreServer interface.
func (s *BucketStore) Series(req *storepb.SeriesRequest, seriesSrv storepb.Store_SeriesServer) (err error) {
	srv := newFlushableServer(
		newBatchableServer(seriesSrv, int(req.ResponseBatchSize)),
		sortingStrategyNone)

	if s.queryGate != nil {
		tracing.DoInSpan(srv.Context(), "store_query_gate_ismyturn", func(ctx context.Context) {
			err = s.queryGate.Start(srv.Context())
		})
		if err != nil {
			return errors.Wrapf(err, "failed to wait for turn")
		}

		defer s.queryGate.Done()
	}

	tenant, _ := tenancy.GetTenantFromGRPCMetadata(srv.Context())

	matchers, err := storecache.MatchersToPromMatchersCached(s.matcherCache, req.Matchers...)
	if err != nil {
		return status.Error(codes.InvalidArgument, err.Error())
	}
	req.MinTime = s.limitMinTime(req.MinTime)
	req.MaxTime = s.limitMaxTime(req.MaxTime)

	var (
		bytesLimiter     = s.bytesLimiterFactory(s.metrics.queriesDropped.WithLabelValues("bytes", tenant))
		ctx              = srv.Context()
		stats            = &queryStats{}

View on GitHub (pinned to 35b8b99117)