thanos-io/thanos · error

proxy TSDBStatistics

Error message

proxy TSDBStatistics

What it means

The querier/gateway TSDBStatistics fan-out calls the wrapped gossip/proxy (`gc.proxy.TSDBStatistics`) which aggregates responses from all peers; any error from the proxy is wrapped as 'proxy TSDBStatistics'. It means the distributed statistics aggregation itself failed (e.g. one member's stream errored under ABORT strategy).

Solutions

1. Inspect the wrapped cause — usually a failed store/member stream (see underlying error).
2. Use PartialResponseStrategy WARNINGS so a single unhealthy peer doesn't fail the aggregation.
3. Restore the failed store node or remove it from the peer set.
4. Increase request timeouts for clusters with many stores.

Example fix

// before
req := &statuspb.TSDBStatisticsRequest{PartialResponseStrategy: storepb.PartialResponseStrategy_ABORT}
// after
req := &statuspb.TSDBStatisticsRequest{PartialResponseStrategy: storepb.PartialResponseStrategy_WARNINGS}

res, warnings, err := api.TSDBStatistics(ctx, req)
if err != nil {
    if strings.Contains(err.Error(), "proxy TSDBStatistics") {
        // fall back to per-store direct queries skipping bad peers
        return collectDirectFromHealthyStores(ctx)
    }
    return err
}

When it happens

Trigger: Calling (Store|Bucket) TSDBStatistics API which invokes `gc.proxy.TSDBStatistics(req, srv)` and receives an error — typically propagated from a member/store stream failure like errors 922-927.

Common situations: One store node down while using ABORT partial response strategy; peer-to-peer proxy connection failure in store-gateway clusters; request timeouts during large aggregations.

Related errors

• proxy Exemplars
• proxy MetricMetadata
• fetching metric metadata from metadata client
• proxy Rules
• fetching tsdb statistics from status client

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/c9cbb53e54bc4d85. Report an issue: GitHub.

Appendix: source

Thrown at pkg/status/status.go:48

	proxy statuspb.StatusServer
}

func NewGRPCClient(ss statuspb.StatusServer) *GRPCClient {
	return &GRPCClient{
		proxy: ss,
	}
}

func (gc *GRPCClient) TSDBStatistics(ctx context.Context, req *statuspb.TSDBStatisticsRequest) (map[string]*statuspb.TSDBStatisticsEntry, annotations.Annotations, error) {
	span, ctx := tracing.StartSpan(ctx, "tsdb_statistics_grpc_request")
	defer span.Finish()

	srv := &tsdbStatisticsServer{
		ctx:            ctx,
		tsdbStatistics: map[string][]*statuspb.TSDBStatisticsEntry{},
	}
	if err := gc.proxy.TSDBStatistics(req, srv); err != nil {
		return nil, nil, errors.Wrap(err, "proxy TSDBStatistics")
	}

	// Collect and aggregate the results.
	tsdbStatistics := make(map[string]*statuspb.TSDBStatisticsEntry)
	for tenant, stats := range srv.tsdbStatistics {
		tenantStats, found := tsdbStatistics[tenant]
		if !found {
			tenantStats = &statuspb.TSDBStatisticsEntry{}
			tsdbStatistics[tenant] = tenantStats
		}

		for _, stat := range stats {
			tenantStats.Merge(stat)
		}
	}

	return tsdbStatistics, srv.warnings, nil
}

View on GitHub (pinned to 35b8b99117)