thanos-io/thanos · error

fetching tsdb statistics from status client

Error message

fetching tsdb statistics from status client %v

What it means

The status proxy's receive goroutine opens a TSDBStatistics stream to a store client; if that RPC fails to start, the error is wrapped with the client's address. Under PartialResponseStrategy_ABORT it is a fatal error; otherwise it is downgraded to a warning response sent to the caller.

Solutions

1. Check connectivity to the store address shown in the error (the wrapped %v).
2. Restart or replace the unhealthy store node, or remove it from the store set.
3. Switch PartialResponseStrategy to WARNING if partial results are acceptable.
4. Inspect store-side logs for the underlying gRPC error (unavailable, deadline, auth).

Example fix

// before
req.PartialResponseStrategy = storepb.PartialResponseStrategy_ABORT
// after
req.PartialResponseStrategy = storepb.PartialResponseStrategy_WARNINGS // tolerate down stores with partial results

if err != nil {
    if strings.Contains(err.Error(), "fetching tsdb statistics from status client") {
        logger.Warn("store unavailable, continuing with partial stats", "store", storeAddr)
        return partialStats, nil
    }
    return err
}

When it happens

Trigger: Streaming TSDB statistics from a store whose gRPC endpoint is unreachable, the store returns an error, or the per-request timeout fires when `stream.client.TSDBStatistics(ctx, stream.request)` is called.

Common situations: A store node is down or restarted; wrong/DNS-stale store address; TLS misconfiguration between querier and store; store overloaded and rejecting new RPCs; ABORT strategy makes one bad store fail the whole request.

Related errors

• proxy MetricMetadata
• fetching metric metadata from metadata client
• receiving tsdb statistics from status client
• Not ready
• no query API server reachable

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/4d16561d58f5c64a. Report an issue: GitHub.

Appendix: source

Thrown at pkg/status/proxy.go:102

type statsStream struct {
	client  statuspb.StatusClient
	request *statuspb.TSDBStatisticsRequest
	channel chan<- *statuspb.TSDBStatistics
	server  statuspb.Status_TSDBStatisticsServer
}

func (stream *statsStream) receive(ctx context.Context) error {
	var (
		err         error
		statsClient statuspb.Status_TSDBStatisticsClient
	)

	tracing.DoInSpan(ctx, "receive_tsdb_statistics_stream_request", func(ctx context.Context) {
		statsClient, err = stream.client.TSDBStatistics(ctx, stream.request)
	})

	if err != nil {
		err = errors.Wrapf(err, "fetching tsdb statistics from status client %v", stream.client)

		if stream.request.PartialResponseStrategy == storepb.PartialResponseStrategy_ABORT {
			return err
		}

		if serr := stream.server.Send(statuspb.NewWarningTSDBStatisticsResponse(err)); serr != nil {
			return serr
		}

		// Not an error if response strategy is warning.
		return nil
	}

	for {
		resp, err := statsClient.Recv()
		if err == io.EOF {
			return nil
		}

View on GitHub (pinned to 35b8b99117)