thanos-io/thanos · error

initial external labels query

Error message

initial external labels query

What it means

The blocking initial query of Prometheus external labels (GET /api/v1/status/config) failed. The sidecar must fetch external labels once before joining the store/gossip cluster; any client error from that first call is wrapped as 'initial external labels query'.

Solutions

1. Verify curl http://<prometheus-url>/api/v1/status/config works from the sidecar
2. Fix --prometheus.url / DNS / network policies
3. Wait for Prometheus readiness (or add readiness ordering) and restart the sidecar
4. Increase --prometheus.get-config-interval/timeout so retries span a slow Prometheus start

Example fix

// before
--prometheus.get-config-timeout=2s   # too short
// after
--prometheus.get-config-timeout=30s --prometheus.get-config-interval=5s

resp, err := http.Get(promURL + "/api/v1/status/config")
if err != nil || resp.StatusCode != 200 {
    return fmt.Errorf("prometheus config endpoint unreachable")
}

err := runutil.Retry(interval, ctx.Done(), func() error {
    return promClient.Labels(ctx)
})
if err != nil {
    return errors.Wrap(err, "initial external labels query")
}

When it happens

Trigger: errors.Wrap after the retried label fetch in runSidecar: promClient.Labels(ctx) failed — --prometheus.url unreachable, malformed Prometheus config returned by the API, request timeout, or Prometheus returned non-2xx during the loop.

Common situations: Prometheus still starting when the sidecar boots and retries exhaust; network policy blocking sidecar→Prometheus; Prometheus config endpoint returning an unexpected payload.

Understand the failure class

Background: 'Something went wrong' / 'Request failed (500)' / 'HTTP error! status: 404' — what failed HTTP requests actually mean and how to find the real cause — this error's family across 28 libraries.

Related errors

• failed to validate prometheus flags
• failed to create matchers cache
• error starting web server
• parse labels
• failed to get prometheus version

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/3819b2eecde45209. Report an issue: GitHub.

Appendix: source

Thrown at cmd/thanos/sidecar.go:266

					return err
				}

				if err := m.UpdateLabels(iterCtx); err != nil {
					level.Warn(logger).Log(
						"msg", "failed to fetch initial external labels. Is Prometheus running? Retrying",
						"err", err,
					)
					return err
				}

				level.Info(logger).Log(
					"msg", "successfully loaded prometheus external labels",
					"external_labels", m.Labels().String(),
				)
				return nil
			})
			if err != nil {
				return errors.Wrap(err, "initial external labels query")
			}

			if m.Labels().Len() == 0 {
				return errors.New("no external labels configured on Prometheus server, uniquely identifying external labels must be configured; see https://thanos.io/tip/thanos/storage.md#external-labels for details.")
			}
			promUp.Set(1)
			statusProber.Ready()

			close(readyToStartGRPC)

			// Periodically query the Prometheus config. We use this as a heartbeat as well as for updating
			// the external labels we apply.
			return runutil.Repeat(conf.prometheus.getConfigInterval, ctx.Done(), func() error {
				iterCtx, iterCancel := context.WithTimeout(context.Background(), conf.prometheus.getConfigTimeout)
				defer iterCancel()
				if err := m.UpdateTimestamps(iterCtx); err != nil {
					level.Warn(logger).Log("msg", "updating timestamps failed", "err", err)
					promUp.Set(0)

View on GitHub (pinned to 35b8b99117)