thanos-io/thanos · error

failed to get tsdb status from prometheus

Error message

failed to get tsdb status from prometheus

What it means

The Thanos sidecar failed to fetch TSDB status (cardinality stats) from Prometheus via the gRPC/gateway endpoint, and wraps the underlying error with this message. It is thrown inside the TSDBStatus HTTP handler in the sidecar when TSDBStatusInGRPC returns an error, meaning Prometheus was unreachable, returned a non-2xx response, or the status API call failed.

Solutions

1. Check Prometheus is up and --prometheus.url is correct and reachable (curl <prometheus.url>/api/v1/status/tsdb).
2. Inspect the wrapped inner error (connection refused vs HTTP status) to determine if it is network or API-version related.
3. Retry the request; transient failures during Prometheus restarts resolve themselves.
4. Lower the limit parameter to reduce load if the endpoint times out on large TSDBs.

Example fix

// before
stats, err := sidecar.GetTSDBStatus(ctx, url, 100000)
// after
if err != nil {
    level.Warn(logger).Log("msg", "tsdb status unavailable, prometheus unreachable?", "err", err)
    // fall back to cached status or retry with backoff
}

resp, err := http.Get(promURL + "/api/v1/status/tsdb?limit=1")
if err != nil || resp.StatusCode != 200 { /* Prometheus not ready */ }

if err != nil {
    if errors.Is(err, context.DeadlineExceeded) { /* retry with backoff */ }
    level.Warn(logger).Log("msg", "tsdb status fetch failed", "err", err)
}

When it happens

Trigger: A client requests the sidecar's /api/v1/status/tsdb endpoint; the sidecar calls TSDBStatusInGRPC against conf.prometheus.url, and that call returns any error (connection refused, timeout, non-200 from Prometheus, malformed response).

Common situations: Prometheus is down or restarting; the --prometheus.url points to the wrong host/port; a proxy blocks the endpoint; Prometheus is an older version without the v2 status API; network timeouts under load while collecting cardinality stats with a large limit.

Understand the failure class

Background: 'Something went wrong' / 'Request failed (500)' / 'HTTP error! status: 404' — what failed HTTP requests actually mean and how to find the real cause — this error's family across 28 libraries.

Related errors

• no query API server reachable
• failed to validate prometheus flags
• not uploading as no external labels are configured yet - is…
• fetch Prometheus flags
• retrieving targets

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/50de65ba899ec0c4. Report an issue: GitHub.

Appendix: source

Thrown at cmd/thanos/sidecar.go:382

					ctx, cancel := context.WithTimeout(context.Background(), conf.prometheus.getConfigTimeout)
					defer cancel()

					// Check if external labels match the provided matchers.
					extLabels := m.Labels()
					promMatchers, err := storepb.MatchersToPromMatchers(matchers...)
					if err != nil {
						return nil, errors.Wrap(err, "failed to convert matchers")
					}
					for _, matcher := range promMatchers {
						if !matcher.Matches(extLabels.Get(matcher.Name)) {
							// External labels don't match, return empty result.
							return nil, nil
						}
					}

					statsEntry, err := c.TSDBStatusInGRPC(ctx, conf.prometheus.url, limit)
					if err != nil {
						return nil, errors.Wrap(err, "failed to get tsdb status from prometheus")
					}

					return map[string]tsdb.Stats{
						"": statsEntry.ToTSDBStats(limit),
					}, nil
				}),
			),
		)

		storeServer := store.NewLimitedStoreServer(store.NewInstrumentedStoreServer(reg, promStore), reg, conf.storeRateLimits)
		s := grpcserver.New(logger, reg, tracer, grpcLogOpts, logFilterMethods, comp, grpcProbe,
			grpcserver.WithServer(store.RegisterStoreServer(storeServer, logger)),
			grpcserver.WithServer(rules.RegisterRulesServer(rules.NewPrometheus(conf.prometheus.url, c, m.Labels))),
			grpcserver.WithServer(targets.RegisterTargetsServer(targets.NewPrometheus(conf.prometheus.url, c, m.Labels))),
			grpcserver.WithServer(meta.RegisterMetadataServer(meta.NewPrometheus(conf.prometheus.url, c))),
			grpcserver.WithServer(exemplars.RegisterExemplarsServer(exemplarSrv)),
			grpcserver.WithServer(info.RegisterInfoServer(infoSrv)),
			grpcserver.WithServer(status.RegisterStatusServer(statusSrv)),

View on GitHub (pinned to 35b8b99117)