thanos-io/thanos · error

receive series from

Error message

receive series from %s

What it means

The gRPC stream Recv() for a Series call returned an error (non-EOF) while lazily pulling series from a store. This is the generic receive failure — RPC error, stream reset, deadline, or store-side failure — wrapped with the store address for attribution; it surfaces as a warning when partial responses are enabled.

Solutions

1. Check the named store's logs for the RPC failure cause
2. Verify gRPC keepalive, timeouts and connection limits between query and store
3. Retry the query; transient stream resets are common under load
4. Use partial responses to keep results from healthy stores

When it happens

Trigger: Thrown at pkg/store/proxy_merge.go:491 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/c86b520b5156d592. Report an issue: GitHub.

Appendix: source

Thrown at pkg/store/proxy_merge.go:491

			resp, err := cl.Recv()
			if err != nil {
				if err == io.EOF {
					l.bufferedResponsesMtx.Lock()
					l.noMoreData = true
					l.dataOrFinishEvent.Signal()
					l.bufferedResponsesMtx.Unlock()
					return false
				}

				var rerr error
				// If timer is already stopped
				if t != nil && !t.Stop() {
					if t.C != nil {
						<-t.C // Drain the channel if it was already stopped.
					}
					rerr = errors.Wrapf(err, "failed to receive any data in %s from %s", l.frameTimeout, st)
				} else {
					rerr = errors.Wrapf(err, "receive series from %s", st)
				}
				l.bufferedResponsesMtx.Lock()
				l.rb.append(storepb.NewWarnSeriesResponse(rerr))
				l.noMoreData = true
				l.dataOrFinishEvent.Signal()
				l.bufferedResponsesMtx.Unlock()
				return false
			}
			if t != nil {
				// frameTimeout only applies to cl.Recv() gRPC call because the goroutine may be blocked on waiting for an empty buffer slot.
				// Set the timeout to the largest possible value to avoid triggering it.
				t.Reset(time.Duration(math.MaxInt64))
			}

			numResponses++
			bytesProcessed += resp.Size()

			if resp.GetSeries() != nil && applySharding && !shardMatcher.MatchesZLabels(resp.GetSeries().Labels) {

View on GitHub (pinned to 35b8b99117)