thanos-io/thanos · error

failed to validate prometheus flags

Error message

failed to validate prometheus flags

What it means

The periodic Prometheus flags validation (checking --prometheus.url is reachable and flags such as external-labels/TSDB settings are consistent) returned an error, wrapped as 'failed to validate prometheus flags'. The sidecar retries validation in the background; if a single validation attempt fails, the underlying client error is wrapped here and reported by run.Group.

Solutions

1. Confirm --prometheus.url (scheme/host/port) resolves to the Prometheus HTTP API from the sidecar pod (curl <url>/api/v1/status/buildinfo)
2. Retry the sidecar if Prometheus was temporarily down during startup
3. Remove any proxy that mangles /api/v1 paths or point directly at the Prometheus service
4. Check sidecar→Prometheus network policies/firewall rules

Example fix

// before
--prometheus.url=http://prometheus:9091   # wrong port
// after
--prometheus.url=http://prometheus.monitoring.svc.cluster.local:9090

resp, err := http.Get(promURL + "/api/v1/status/buildinfo")
if err != nil || resp.StatusCode != 200 {
    return fmt.Errorf("prometheus flags endpoint unreachable at %s", promURL)
}

err := runutil.Retry(interval, ctx.Done(), validatePrometheusFlags)
if err != nil {
    level.Warn(logger).Log("msg", "prometheus flags validation failed, will retry", "err", err)
}

When it happens

Trigger: errors.Wrap around the validation goroutine in runSidecar: the inner func returned an error from promClient.BuildVersion / flag checks — typically because --prometheus.url is wrong or unreachable, the endpoint is not a Prometheus server, or a transient HTTP failure occurred in the non-retried validation path.

Common situations: Service DNS name wrong in Kubernetes; Prometheus on a different port than 9090; TLS/proxy in front of Prometheus altering paths; Prometheus down during sidecar startup.

Understand the failure class

Background: 'Something went wrong' / 'Request failed (500)' / 'HTTP error! status: 404' — what failed HTTP requests actually mean and how to find the real cause — this error's family across 28 libraries.

Related errors

• error starting web server
• initial external labels query
• perform request against
• request config against
• read query instant response

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/6033a169a57b40dd. Report an issue: GitHub.

Appendix: source

Thrown at cmd/thanos/sidecar.go:211

				err := runutil.Retry(conf.prometheus.getConfigInterval, ctx.Done(), func() error {
					iterCtx, iterCancel := context.WithTimeout(context.Background(), conf.prometheus.getConfigTimeout)
					defer iterCancel()

					if err := validatePrometheus(iterCtx, m.client, logger, &conf, m); err != nil {
						level.Warn(logger).Log(
							"msg", "failed to validate prometheus flags. Is Prometheus running? Retrying",
							"err", err,
						)
						return err
					}

					level.Info(logger).Log(
						"msg", "successfully validated prometheus flags",
					)
					return nil
				})
				if err != nil {
					return errors.Wrap(err, "failed to validate prometheus flags")
				}
			}

			// We retry infinitely until we reach and fetch BuildVersion from our Prometheus.
			err := runutil.Retry(conf.prometheus.getConfigInterval, ctx.Done(), func() error {
				iterCtx, iterCancel := context.WithTimeout(context.Background(), conf.prometheus.getConfigTimeout)
				defer iterCancel()

				if err := m.BuildVersion(iterCtx); err != nil {
					level.Warn(logger).Log(
						"msg", "failed to fetch prometheus version. Is Prometheus running? Retrying",
						"err", err,
					)
					return err
				}

				level.Info(logger).Log(
					"msg", "successfully loaded prometheus version",

View on GitHub (pinned to 35b8b99117)