{"record":{"id":"fc4923cbc70c2084","repo":"thanos-io/thanos","slug":"failed-to-wait-for-turn-fc4923","errorCode":null,"errorMessage":"failed to wait for turn","messagePattern":"failed to wait for turn","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"pkg/store/bucket.go","lineNumber":1559,"sourceCode":"\t}\n\n\tparts = append(parts, fmt.Sprintf(\"Range: %d-%d Resolution: %d\", currMin, currMax, currRes))\n\n\tlevel.Debug(logger).Log(\"msg\", \"Blocks source resolutions\", \"blocks\", len(bs), \"Maximum Resolution\", maxResolutionMillis, \"mint\", mint, \"maxt\", maxt, \"lset\", lset.String(), \"spans\", strings.Join(parts, \"\\n\"))\n}\n\n// Series implements the storepb.StoreServer interface.\nfunc (s *BucketStore) Series(req *storepb.SeriesRequest, seriesSrv storepb.Store_SeriesServer) (err error) {\n\tsrv := newFlushableServer(\n\t\tnewBatchableServer(seriesSrv, int(req.ResponseBatchSize)),\n\t\tsortingStrategyNone)\n\n\tif s.queryGate != nil {\n\t\ttracing.DoInSpan(srv.Context(), \"store_query_gate_ismyturn\", func(ctx context.Context) {\n\t\t\terr = s.queryGate.Start(srv.Context())\n\t\t})\n\t\tif err != nil {\n\t\t\treturn errors.Wrapf(err, \"failed to wait for turn\")\n\t\t}\n\n\t\tdefer s.queryGate.Done()\n\t}\n\n\ttenant, _ := tenancy.GetTenantFromGRPCMetadata(srv.Context())\n\n\tmatchers, err := storecache.MatchersToPromMatchersCached(s.matcherCache, req.Matchers...)\n\tif err != nil {\n\t\treturn status.Error(codes.InvalidArgument, err.Error())\n\t}\n\treq.MinTime = s.limitMinTime(req.MinTime)\n\treq.MaxTime = s.limitMaxTime(req.MaxTime)\n\n\tvar (\n\t\tbytesLimiter     = s.bytesLimiterFactory(s.metrics.queriesDropped.WithLabelValues(\"bytes\", tenant))\n\t\tctx              = srv.Context()\n\t\tstats            = &queryStats{}","sourceCodeStart":1541,"sourceCodeEnd":1577,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/pkg/store/bucket.go#L1541-L1577","documentation":"The store gateway uses a concurrency-limiting query gate before serving Series requests. This error wraps any failure from queryGate.Start() (waiting for a turn to execute) during the gRPC Series handler. Typically the underlying cause is the caller's context being canceled (client disconnect/timeout) or the gate's limiter rejecting entry while shutting down.","triggerScenarios":"A gRPC Series call arrives while the store's query gate is saturated or the store is shutting down; the client context is canceled before a concurrency slot becomes free, so queryGate.Start(srv.Context()) returns context.Canceled/DeadlineExceeded.","commonSituations":"Query fan-out over many blocks exceeding the store's configured concurrency; querier timeouts shorter than the gate wait time; client cancellation mid-query; store restart under load.","solutions":["Check the wrapped cause (context canceled vs deadline exceeded) in the error chain to distinguish client disconnects from overload.","Increase --store.grpc.series-sample-concurrency (or the query gate limit) or reduce query fan-out to shorten wait times.","Raise querier/store timeout values so queries are not killed while queued.","Retry idempotent queries on DeadlineExceeded; investigate load if the gate is persistently saturated."],"exampleFix":"// querier side: tolerate transient gate waits\ncctx, cancel := context.WithTimeout(ctx, 2*time.Minute)\ndefer cancel()\nseries, err := storeClient.Series(cctx, req)\nif errors.Is(err, context.DeadlineExceeded) {\n    series, err = storeClient.Series(ctx, req) // retry once\n}","handlingStrategy":"retry","validationCode":"// before calling, check gate saturation if exposed\nif storeGate != nil && storeGate.IsBlocked() { /* wait or back off before issuing the call */ }","typeGuard":null,"tryCatchPattern":"err := s.queryGate.Start(ctx)\nif err != nil {\n    if errors.Is(err, context.Canceled) {\n        return status.Error(codes.Canceled, \"client canceled while waiting for turn\")\n    }\n    if errors.Is(err, context.DeadlineExceeded) {\n        return status.Error(codes.DeadlineExceeded, \"timed out waiting for query gate\")\n    }\n    return status.Error(codes.Unavailable, err.Error())\n}","preventionTips":["Size query-gate concurrency to expected query fan-out.","Set client timeouts larger than worst-case gate wait.","Monitor gate wait metrics and alert on sustained saturation.","Retry only idempotent Series calls on DeadlineExceeded."],"tags":["thanos","grpc","concurrency","rate-limiting"],"backgroundTag":"request-timeout","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}