thanos-io/thanos · error · ApiError

retrieving tsdb statistics

Error message

retrieving tsdb statistics

What it means

Wraps failures of the status service's TSDBStatistics gRPC call (cardinality analysis) into an ErrorInternal for /api/v1/tsdb/status. The RPC failed or timed out while collecting per-tenant statistics.

Solutions

1. Retry with a smaller match[] selector and/or lower limit to reduce scan cost.
2. Check store gateway health/logs for OOM or timeouts during TSDB statistics collection.
3. Increase request/context timeouts and verify querier-store connectivity.

const resp = await fetch(`${base}/api/v1/tsdb/status?limit=1000`);
if (!resp.ok) {
  const body = await resp.json();
  if (body.errorType === 'internal') {
    // narrow match[] selector and retry with backoff
    return retryWithSmallerSelector();
  }
  throw new Error(body.error);
}

When it happens

Trigger: Calling /api/v1/tsdb/status when the underlying status service/store RPC errors: store unreachable, context deadline exceeded during expensive cardinality scans, or store returning an internal error.

Common situations: Large Prometheus TSDBs causing cardinality scans to exceed timeouts; store gateway OOMs or restarts under the head-stats query; network issues between querier and stores.

Related errors

• retrieving metadata
• building gRPC client
• endpoint
• setup gRPC server
• Not ready

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/23fe22fb526042ff. Report an issue: GitHub.

Appendix: source

Thrown at pkg/api/query/v1.go:1762

		}
	}

	if limit > math.MaxInt32 {
		return nil, nil, &api.ApiError{Typ: api.ErrorBadData, Err: errors.Errorf("limit %d overflows int32", limit)}, func() {}
	}
	req := &statuspb.TSDBStatisticsRequest{
		Matchers:                matchers,
		Limit:                   int32(limit),
		PartialResponseStrategy: ps,
	}

	var stats map[string]*statuspb.TSDBStatisticsEntry
	tracing.DoInSpan(ctx, "retrieve_tsdb_statistics", func(ctx context.Context) {
		stats, warnings, err = qapi.status.TSDBStatistics(ctx, req)
	})

	if err != nil {
		return nil, nil, &api.ApiError{Typ: api.ErrorInternal, Err: errors.Wrap(err, "retrieving tsdb statistics")}, func() {}
	}

	if tenant != "" {
		if stats[tenant] == nil {
			return nil, nil, &api.ApiError{Typ: api.ErrorInternal, Err: errors.Wrap(fmt.Errorf("%s tenant not found", tenant), "retrieving tsdb statistics")}, func() {}
		}

		return convertToTSDBSTatus(stats[tenant], limit), warnings.AsErrors(), nil, func() {}
	}

	// Merge statistics from all tenants.
	aggregatedStats := &statuspb.TSDBStatisticsEntry{}
	for _, v := range stats {
		aggregatedStats.Merge(v)
	}

	return convertToTSDBSTatus(aggregatedStats, limit), warnings.AsErrors(), nil, func() {}
}

View on GitHub (pinned to 35b8b99117)