thanos-io/thanos · warning

not uploading as no external labels are configured yet - is…

Error message

not uploading as no external labels are configured yet - is Prometheus healthy/reachable?

What it means

This error is returned (and retried) inside the sidecar's main run loop when the Prometheus external-labels provider reports zero labels. Thanos requires external labels (e.g. prometheus=<cluster>) to stamp blocks before uploading to object storage, so it keeps retrying for promReadyTimeout and fails if Prometheus never provides them.

Solutions

1. Add an external_labels section (e.g. external_labels: {prometheus: <name>, replica: <n>}) to prometheus.yml and reload Prometheus.
2. Verify the sidecar's --prometheus.url points at the correct Prometheus instance.
3. Increase --prometheus.ready-timeout if Prometheus legitimately takes long to become ready.
4. Check Prometheus /-/healthy and that the labels endpoint responds before the timeout elapses.

Example fix

# before (prometheus.yml)
global:
  scrape_interval: 15s
# after
global:
  scrape_interval: 15s
  external_labels:
    prometheus: eu1-prod
    replica: 0

curl -s <promURL>/api/v1/status/config | grep -A3 external_labels

When it happens

Trigger: m.Labels().Len() == 0 at the start of sidecar operation: Prometheus has no external_labels configured in its config, or the sidecar cannot yet scrape/fetch them from the configured Prometheus URL within the retry window.

Common situations: Fresh Prometheus install without external_labels section; sidecar started before Prometheus finished loading config; misconfigured --prometheus.url so labels endpoint queries the wrong instance; Prometheus not yet healthy after startup.

Understand the failure class

Background: "X is required", "must be set", "cannot be empty": the missing-required-config error family, from Vertex AI project/location to WeChat keys — this error's family across 18 libraries.

Related errors

• failed to create matchers cache
• failed to get tsdb status from prometheus
• aborting as no external labels found after waiting
• opening tsdb directory
• found that Prometheus and Thanos use different paths for…

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/e25ffde956f7bc0a. Report an issue: GitHub.

Appendix: source

Thrown at cmd/thanos/sidecar.go:453

				runutil.CloseWithLogOnErr(logger, bkt, "bucket client")
			}
		}()

		if err := promclient.IsWALDirAccessible(conf.tsdb.path); err != nil {
			level.Error(logger).Log("err", err)
		}

		ctx, cancel := context.WithCancel(context.Background())
		g.Add(func() error {
			defer runutil.CloseWithLogOnErr(logger, bkt, "bucket client")

			promReadyTimeout := conf.prometheus.readyTimeout
			extLabelsCtx, cancel := context.WithTimeout(ctx, promReadyTimeout)
			defer cancel()

			if err := runutil.Retry(2*time.Second, extLabelsCtx.Done(), func() error {
				if m.Labels().Len() == 0 {
					return errors.New("not uploading as no external labels are configured yet - is Prometheus healthy/reachable?")
				}
				return nil
			}); err != nil {
				return errors.Wrapf(err, "aborting as no external labels found after waiting %s", promReadyTimeout)
			}

			dataDir, err := os.OpenRoot(conf.tsdb.path)
			if err != nil {
				return errors.Wrap(err, "opening tsdb directory")
			}
			defer runutil.CloseWithLogOnErr(logger, dataDir, "tsdb directory")

			s := shipper.New(
				bkt,
				dataDir,
				shipper.WithLogger(logger),
				shipper.WithRegisterer(reg),
				shipper.WithSource(metadata.SidecarSource),

View on GitHub (pinned to 35b8b99117)