thanos-io/thanos · error

receiving tsdb statistics from status client

Error message

receiving tsdb statistics from status client %v

What it means

While consuming the store's statistics stream (`statsClient.Recv()`), any error other than io.EOF is wrapped as 'receiving tsdb statistics from status client %v'. It indicates the upstream stream broke mid-transfer rather than failing to start.

Solutions

1. Retry the statistics request; transient stream drops are common.
2. Check the store node's logs and health at the time of the failure.
3. Adjust proxy/LB idle timeouts to exceed the expected streaming duration.
4. Use PartialResponseStrategy WARNINGS to surface this as a warning instead of failing the request.

Example fix

// before
if err != nil { return err } // aborts whole request on ABORT strategy
// after
if err != nil {
    logger.Warn("tsdb statistics stream from store broke, results partial", "err", err)
    return nil // with WARNINGS strategy
}

err := streamStats(ctx)
if err != nil && strings.Contains(err.Error(), "receiving tsdb statistics") {
    // transient mid-stream drop: retry whole request
    return retryWithBackoff(streamStats, ctx)
}

When it happens

Trigger: Calling receive() on the proxy stream when the store's gRPC connection drops, the server cancels the stream, or a transport/deadline error occurs after the initial TSDBStatistics call succeeded.

Common situations: Store crashes mid-response; connection reset by load balancer idle timeout on long streams; context cancellation by the downstream client; TLS handshake re-negotiation failure.

Related errors

• proxy MetricMetadata
• fetching metric metadata from metadata client
• receiving metric metadata from metadata client
• sending rules warning to server
• Unknown

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/124a20a5fa413a43. Report an issue: GitHub.

Appendix: source

Thrown at pkg/status/proxy.go:123

			return err
		}

		if serr := stream.server.Send(statuspb.NewWarningTSDBStatisticsResponse(err)); serr != nil {
			return serr
		}

		// Not an error if response strategy is warning.
		return nil
	}

	for {
		resp, err := statsClient.Recv()
		if err == io.EOF {
			return nil
		}

		if err != nil {
			err = errors.Wrapf(err, "receiving tsdb statistics from status client %v", stream.client)

			if stream.request.PartialResponseStrategy == storepb.PartialResponseStrategy_ABORT {
				return err
			}

			if err := stream.server.Send(statuspb.NewWarningTSDBStatisticsResponse(err)); err != nil {
				return errors.Wrapf(err, "sending tsdb statistics error to server %v", stream.server)
			}

			// Not an error if response strategy is warning.
			return nil
		}

		if w := resp.GetWarning(); w != "" {
			if err := stream.server.Send(statuspb.NewWarningTSDBStatisticsResponse(errors.New(w))); err != nil {
				return errors.Wrapf(err, "sending tsdb statistics warning to server %v", stream.server)
			}
			continue

View on GitHub (pinned to 35b8b99117)