thanos-io/thanos · error

failed to get prometheus version

Error message

failed to get prometheus version

What it means

The sidecar could not fetch the Prometheus build version via the /api/v1/status/buildinfo endpoint. runutil.Retry keeps retrying until success or ctx cancellation; if the context ends or the final attempt fails, the error is wrapped as 'failed to get prometheus version'. The version is needed to adapt API behavior (e.g. flags endpoint availability).

Solutions

1. Check the endpoint manually: curl http://<prometheus-url>/api/v1/status/buildinfo
2. Fix --prometheus.url and confirm DNS/network reachability from the sidecar
3. Upgrade or reconfigure the proxy to expose the buildinfo endpoint (requires Prometheus 2.x)
4. Increase --prometheus.get-config-timeout if the endpoint is slow, or investigate why Prometheus is down

Example fix

// before
iterCtx timeout 5s against slow/auth-blocked endpoint -> loop fails
// after
--prometheus.get-config-timeout=30s
# plus allow /api/v1/status/buildinfo in the auth proxy config

resp, err := http.Get(promURL + "/api/v1/status/buildinfo")
if err != nil {
    return err
}
if resp.StatusCode != 200 {
    return fmt.Errorf("buildinfo returned %d", resp.StatusCode)
}

err := runutil.Retry(conf.prometheus.getConfigInterval, ctx.Done(), func() error {
    return promClient.BuildVersion(iterCtx)
})
if err != nil {
    return errors.Wrap(err, "failed to get prometheus version")
}

When it happens

Trigger: errors.Wrap after the retry loop in runSidecar: promClient.BuildVersionNative(iterCtx) failed on every retry or the context was cancelled — wrong --prometheus.url, Prometheus not exposing buildinfo (proxies, very old Prometheus versions without the endpoint), timeouts.

Common situations: Prometheus behind an auth proxy that blocks /api/v1/status/buildinfo; Prometheus 1.x or stripped-down builds lacking the endpoint; sidecar killed before Prometheus became ready; DNS misconfiguration.

Understand the failure class

Background: 'Something went wrong' / 'Request failed (500)' / 'HTTP error! status: 404' — what failed HTTP requests actually mean and how to find the real cause — this error's family across 28 libraries.

Related errors

• failed to validate prometheus flags
• initial external labels query
• fetch Prometheus flags
• error marshaling response
• Admin operations are disabled

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/292d4519d9ee0c91. Report an issue: GitHub.

Appendix: source

Thrown at cmd/thanos/sidecar.go:234

			err := runutil.Retry(conf.prometheus.getConfigInterval, ctx.Done(), func() error {
				iterCtx, iterCancel := context.WithTimeout(context.Background(), conf.prometheus.getConfigTimeout)
				defer iterCancel()

				if err := m.BuildVersion(iterCtx); err != nil {
					level.Warn(logger).Log(
						"msg", "failed to fetch prometheus version. Is Prometheus running? Retrying",
						"err", err,
					)
					return err
				}

				level.Info(logger).Log(
					"msg", "successfully loaded prometheus version",
				)
				return nil
			})
			if err != nil {
				return errors.Wrap(err, "failed to get prometheus version")
			}

			// Blocking query of external labels before joining as a Source Peer into gossip.
			// We retry infinitely until we reach and fetch labels from our Prometheus.
			err = runutil.Retry(conf.prometheus.getConfigInterval, ctx.Done(), func() error {
				iterCtx, iterCancel := context.WithTimeout(context.Background(), conf.prometheus.getConfigTimeout)
				defer iterCancel()

				if err := m.UpdateTimestamps(iterCtx); err != nil {
					level.Warn(logger).Log(
						"msg", "failed to fetch timestamps. Is Prometheus running? Retrying",
						"err", err,
					)
					return err
				}

				if err := m.UpdateLabels(iterCtx); err != nil {
					level.Warn(logger).Log(

View on GitHub (pinned to 35b8b99117)