thanos-io/thanos · error

failed to receive any data in

Error message

failed to receive any data in %s from %s

What it means

The lazy series receiver got a Recv error from a store AND the frame-timeout watchdog timer had already fired, meaning no data frame arrived from that store within frameTimeout. It indicates a stalled or too-slow store stream rather than an immediate RPC failure; the wrapped error is reported as a warning under partial-response mode.

Solutions

1. Increase the store API client frame timeout on the querier
2. Check the failing store for CPU/IO saturation or slow object-store reads
3. Check network latency between query and store components
4. Rerun the query when the store is less loaded; rely on partial responses meanwhile

When it happens

Trigger: Thrown at pkg/store/proxy_merge.go:489 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/e5d65a6e47e1abe6. Report an issue: GitHub.

Appendix: source

Thrown at pkg/store/proxy_merge.go:489

			}

			resp, err := cl.Recv()
			if err != nil {
				if err == io.EOF {
					l.bufferedResponsesMtx.Lock()
					l.noMoreData = true
					l.dataOrFinishEvent.Signal()
					l.bufferedResponsesMtx.Unlock()
					return false
				}

				var rerr error
				// If timer is already stopped
				if t != nil && !t.Stop() {
					if t.C != nil {
						<-t.C // Drain the channel if it was already stopped.
					}
					rerr = errors.Wrapf(err, "failed to receive any data in %s from %s", l.frameTimeout, st)
				} else {
					rerr = errors.Wrapf(err, "receive series from %s", st)
				}
				l.bufferedResponsesMtx.Lock()
				l.rb.append(storepb.NewWarnSeriesResponse(rerr))
				l.noMoreData = true
				l.dataOrFinishEvent.Signal()
				l.bufferedResponsesMtx.Unlock()
				return false
			}
			if t != nil {
				// frameTimeout only applies to cl.Recv() gRPC call because the goroutine may be blocked on waiting for an empty buffer slot.
				// Set the timeout to the largest possible value to avoid triggering it.
				t.Reset(time.Duration(math.MaxInt64))
			}

			numResponses++
			bytesProcessed += resp.Size()

View on GitHub (pinned to 35b8b99117)