thanos-io/thanos · critical

no external labels configured on Prometheus server…

Error message

no external labels configured on Prometheus server, uniquely identifying external labels must be configured; see https://thanos.io/tip/thanos/storage.md#external-labels for details.

What it means

The sidecar requires Prometheus to have uniquely identifying external labels configured; if the fetched external labels set is empty, it returns this hard error and refuses to start, because blocks uploaded by this sidecar would otherwise be indistinguishable from other Prometheus instances.

Solutions

1. Add external_labels to the Prometheus configuration, e.g. external_labels: {cluster: <name>, replica: "0"}, then reload Prometheus and restart the sidecar
2. Ensure external_labels are set globally in prometheus.yml, not only on scrape configs
3. If this Prometheus is not meant for Thanos, stop pointing the sidecar at it
4. Verify labels were actually applied: curl /api/v1/status/config and check the global external_labels section

Example fix

// before
# prometheus.yml
global:
  scrape_interval: 15s
// after
# prometheus.yml
global:
  scrape_interval: 15s
  external_labels:
    cluster: eu1
    replica: "0"

cfg := loadPrometheusConfig()
if len(cfg.Global.ExternalLabels.Map()) == 0 {
    return errors.New("prometheus external_labels must be configured for thanos sidecar")
}

if err := run(); err != nil {
    if strings.Contains(err.Error(), "no external labels") {
        level.Error(logger).Log("msg", "fix prometheus.yml: set global external_labels (cluster, replica)")
    }
    return err
}

When it happens

Trigger: errors.New in runSidecar right after the successful initial external labels query: m.Labels().Len() == 0, i.e. the Prometheus server has no external_labels (or only labels Thanos drops, e.g. replica/promise labels handling differs across versions).

Common situations: Fresh Prometheus deployment without external_labels; global.external_labels removed in a Helm values refactor; users pointing the sidecar at a Prometheus not meant for Thanos.

Understand the failure class

Background: "X is required", "must be set", "cannot be empty": the missing-required-config error family, from Vertex AI project/location to WeChat keys — this error's family across 18 libraries.

Related errors

• unrecognized label
• unsupported format for label
• unquote label value
• unable to load config content
• unable to unmarshal config content

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/47917a2cf4ee4df8. Report an issue: GitHub.

Appendix: source

Thrown at cmd/thanos/sidecar.go:270

					level.Warn(logger).Log(
						"msg", "failed to fetch initial external labels. Is Prometheus running? Retrying",
						"err", err,
					)
					return err
				}

				level.Info(logger).Log(
					"msg", "successfully loaded prometheus external labels",
					"external_labels", m.Labels().String(),
				)
				return nil
			})
			if err != nil {
				return errors.Wrap(err, "initial external labels query")
			}

			if m.Labels().Len() == 0 {
				return errors.New("no external labels configured on Prometheus server, uniquely identifying external labels must be configured; see https://thanos.io/tip/thanos/storage.md#external-labels for details.")
			}
			promUp.Set(1)
			statusProber.Ready()

			close(readyToStartGRPC)

			// Periodically query the Prometheus config. We use this as a heartbeat as well as for updating
			// the external labels we apply.
			return runutil.Repeat(conf.prometheus.getConfigInterval, ctx.Done(), func() error {
				iterCtx, iterCancel := context.WithTimeout(context.Background(), conf.prometheus.getConfigTimeout)
				defer iterCancel()
				if err := m.UpdateTimestamps(iterCtx); err != nil {
					level.Warn(logger).Log("msg", "updating timestamps failed", "err", err)
					promUp.Set(0)
					statusProber.NotReady(err)
					return nil
				}

View on GitHub (pinned to 35b8b99117)